[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 475472486 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002383] x2apic enabled [ 0.003013] Switched APIC routing to physical x2apic. [ 0.004020] kvm-guest: setup PV IPIs [ 0.007317] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008027] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009018] pid_max: default: 32768 minimum: 301 [ 0.010161] LSM: Security Framework initializing [ 0.011068] Yama: becoming mindful. [ 0.012050] SELinux: Initializing. [ 0.013104] *** VALIDATE selinux *** [ 0.024297] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029277] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030150] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031142] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032128] *** VALIDATE tmpfs *** [ 0.034179] *** VALIDATE proc *** [ 0.035328] *** VALIDATE cgroup *** [ 0.036011] *** VALIDATE cgroup2 *** [ 0.037223] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038238] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040028] Spectre V2 : User space: Vulnerable [ 0.041008] Speculative Store Bypass: Vulnerable [ 0.043776] debug: unmapping init [mem 0xffffffff99459000-0xffffffff99460fff] [ 0.045183] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046744] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047021] ... version: 2 [ 0.048012] ... bit width: 48 [ 0.048863] ... generic registers: 4 [ 0.049010] ... value mask: 0000ffffffffffff [ 0.050011] ... max period: 00007fffffffffff [ 0.051017] ... fixed-purpose events: 3 [ 0.052009] ... event mask: 000000070000000f [ 0.053286] rcu: Hierarchical SRCU implementation. [ 0.055500] smp: Bringing up secondary CPUs ... [ 0.056597] x86: Booting SMP configuration: [ 0.057032] .... node #0, CPUs: #1 #2 #3 [ 0.060073] smp: Brought up 1 node, 4 CPUs [ 0.061977] smpboot: Max logical packages: 1 [ 0.062010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.139910] node 0 deferred pages initialised in 76ms [ 0.143122] devtmpfs: initialized [ 0.144241] x86/mm: Memory block size: 128MB [ 0.146678] gcov: version magic: 0x41383552 [ 0.148327] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.149102] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.150252] pinctrl core: initialized pinctrl subsystem [ 0.151202] [ 0.151802] ************************************************************* [ 0.152019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.153012] ** ** [ 0.154014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.155014] ** ** [ 0.156015] ** This means that this kernel is built to expose internal ** [ 0.157013] ** IOMMU data structures, which may compromise security on ** [ 0.158014] ** your system. ** [ 0.159015] ** ** [ 0.160014] ** If you see this message and you are not debugging the ** [ 0.161011] ** kernel, report this immediately to your vendor! ** [ 0.162011] ** ** [ 0.163016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.164013] ************************************************************* [ 0.165705] NET: Registered protocol family 16 [ 0.166468] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.167065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.168064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.170059] cpuidle: using governor menu [ 0.171751] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.174474] PCI: Using configuration type 1 for base access [ 0.176122] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.185102] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.186039] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.188148] cryptd: max_cpu_qlen set to 1000 [ 0.191206] ACPI: Added _OSI(Module Device) [ 0.193017] ACPI: Added _OSI(Processor Device) [ 0.194028] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.196012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.202627] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.209455] ACPI: Interpreter enabled [ 0.211075] ACPI: PM: (supports S0 S3 S4 S5) [ 0.213023] ACPI: Using IOAPIC for interrupt routing [ 0.214108] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.218447] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.229708] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.231044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.235028] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.240140] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.246451] acpiphp: Slot [2] registered [ 0.248241] acpiphp: Slot [5] registered [ 0.250124] acpiphp: Slot [6] registered [ 0.252167] acpiphp: Slot [7] registered [ 0.254161] acpiphp: Slot [8] registered [ 0.256187] acpiphp: Slot [9] registered [ 0.258126] acpiphp: Slot [10] registered [ 0.260135] acpiphp: Slot [3] registered [ 0.262109] acpiphp: Slot [4] registered [ 0.264078] acpiphp: Slot [11] registered [ 0.266105] acpiphp: Slot [12] registered [ 0.267070] acpiphp: Slot [13] registered [ 0.268121] acpiphp: Slot [14] registered [ 0.270133] acpiphp: Slot [15] registered [ 0.271102] acpiphp: Slot [16] registered [ 0.273109] acpiphp: Slot [17] registered [ 0.274092] acpiphp: Slot [18] registered [ 0.276104] acpiphp: Slot [19] registered [ 0.277092] acpiphp: Slot [20] registered [ 0.279117] acpiphp: Slot [21] registered [ 0.280100] acpiphp: Slot [22] registered [ 0.281128] acpiphp: Slot [23] registered [ 0.283184] acpiphp: Slot [24] registered [ 0.285116] acpiphp: Slot [25] registered [ 0.286122] acpiphp: Slot [26] registered [ 0.288111] acpiphp: Slot [27] registered [ 0.289119] acpiphp: Slot [28] registered [ 0.290135] acpiphp: Slot [29] registered [ 0.292113] acpiphp: Slot [30] registered [ 0.293077] acpiphp: Slot [31] registered [ 0.294066] PCI host bridge to bus 0000:00 [ 0.295023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.298023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.300023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.303024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.306024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.308028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.310229] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.313955] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.317454] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.330827] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.335099] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.338019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.340018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.343017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.346759] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.349711] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.354046] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.357960] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.362014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.375016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.380093] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.385471] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.391016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.397013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.413019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.428466] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.435013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.441013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.458015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.467969] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.476028] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.484098] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.503017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.512000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.520023] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.527014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.550014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.561698] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.566015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.574014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.593024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.602417] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.614015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.626013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.641015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.652592] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.657516] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.663404] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.665428] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.667267] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.672106] iommu: Default domain type: Passthrough [ 0.674467] SCSI subsystem initialized [ 0.675115] ACPI: bus type USB registered [ 0.677117] usbcore: registered new interface driver usbfs [ 0.679071] usbcore: registered new interface driver hub [ 0.680061] usbcore: registered new device driver usb [ 0.681126] pps_core: LinuxPPS API ver. 1 registered [ 0.683009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.685065] PTP clock support registered [ 0.688049] EDAC MC: Ver: 3.0.0 [ 0.690154] PCI: Using ACPI for IRQ routing [ 0.691902] NetLabel: Initializing [ 0.694011] NetLabel: domain hash size = 128 [ 0.695010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.697078] NetLabel: unlabeled traffic allowed by default [ 0.701085] vgaarb: loaded [ 0.702274] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.704015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.710524] clocksource: Switched to clocksource kvm-clock [ 0.824042] VFS: Disk quotas dquot_6.6.0 [ 0.825547] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.828259] *** VALIDATE ramfs *** [ 0.829480] *** VALIDATE hugetlbfs *** [ 0.831076] pnp: PnP ACPI init [ 0.833564] pnp: PnP ACPI: found 6 devices [ 0.852265] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.855396] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.857198] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.859452] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.861881] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.864315] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.867338] NET: Registered protocol family 2 [ 0.869448] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.877392] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.881695] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.887805] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.891737] TCP: Hash tables configured (established 65536 bind 65536) [ 0.895121] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.901252] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.904276] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.907707] NET: Registered protocol family 1 [ 0.910634] RPC: Registered named UNIX socket transport module. [ 0.912858] RPC: Registered udp transport module. [ 0.914987] RPC: Registered tcp transport module. [ 0.916912] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.919601] NET: Registered protocol family 44 [ 0.921527] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.923789] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.928795] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.930753] PCI: CLS 0 bytes, default 64 [ 0.932520] Unpacking initramfs... [ 2.326141] debug: unmapping init [mem 0xffff954afcc54000-0xffff954afffbffff] [ 2.330410] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.332775] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.335507] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.828697] Initialise system trusted keyrings [ 2.831086] Key type blacklist registered [ 2.833415] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.843060] zbud: loaded [ 2.846192] *** VALIDATE nfs *** [ 2.847587] *** VALIDATE nfs4 *** [ 2.849388] pstore: using deflate compression [ 2.854374] Platform Keyring initialized [ 2.957363] NET: Registered protocol family 38 [ 2.959315] Key type asymmetric registered [ 2.960459] Asymmetric key parser 'x509' registered [ 2.962561] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.966130] io scheduler mq-deadline registered [ 2.967415] io scheduler kyber registered [ 2.968787] io scheduler bfq registered [ 2.970410] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.973150] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.975700] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.978299] ACPI: Power Button [PWRF] [ 2.985447] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.991481] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.002707] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.009422] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.022645] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.050778] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.078378] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.082112] Non-volatile memory driver v1.3 [ 3.083248] Linux agpgart interface v0.103 [ 3.111102] virtio_blk virtio1: [vda] 147160 512-byte logical blocks (75.3 MB/71.9 MiB) [ 3.113861] vda: detected capacity change from 0 to 75345920 [ 3.131750] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.136102] vdb: detected capacity change from 0 to 1073741824 [ 3.152635] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.155272] vdc: detected capacity change from 0 to 2621440000 [ 3.173663] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.177144] vdd: detected capacity change from 0 to 2621440000 [ 3.202693] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.208303] vde: detected capacity change from 0 to 4294967296 [ 3.226921] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.230258] vdf: detected capacity change from 0 to 4294967296 [ 3.249332] libphy: Fixed MDIO Bus: probed [ 3.254732] usbcore: registered new interface driver usbserial_generic [ 3.257498] usbserial: USB Serial support registered for generic [ 3.260256] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.264647] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.266716] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.269190] mousedev: PS/2 mouse device common for all mice [ 3.272775] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.274518] rtc_cmos 00:05: RTC can wake from S4 [ 3.279380] rtc_cmos 00:05: registered as rtc0 [ 3.279901] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.281183] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.290147] intel_pstate: CPU model not supported [ 3.291839] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.296077] hid: raw HID events driver (C) Jiri Kosina [ 3.298653] usbcore: registered new interface driver usbhid [ 3.300855] usbhid: USB HID core driver [ 3.302851] drop_monitor: Initializing network drop monitor service [ 3.305741] Initializing XFRM netlink socket [ 3.308245] NET: Registered protocol family 10 [ 3.310877] Segment Routing with IPv6 [ 3.312719] NET: Registered protocol family 17 [ 3.315622] mpls_gso: MPLS GSO support [ 3.322176] RAS: Correctable Errors collector initialized. [ 3.325329] AVX version of gcm_enc/dec engaged. [ 3.327138] AES CTR mode by8 optimization enabled [ 3.424669] sched_clock: Marking stable (3424634606, 0)->(4334190026, -909555420) [ 3.429489] registered taskstats version 1 [ 3.432875] Loading compiled-in X.509 certificates [ 3.435676] zswap: loaded using pool lzo/zbud [ 3.464288] Key type big_key registered [ 3.478118] Key type encrypted registered [ 3.479785] ima: No TPM chip found, activating TPM-bypass! [ 3.481938] ima: Allocated hash algorithm: sha1 [ 3.483670] ima: No architecture policies found [ 3.485429] evm: Initialising EVM extended attributes: [ 3.487339] evm: security.selinux [ 3.488494] evm: security.ima [ 3.489465] evm: security.capability [ 3.490738] evm: HMAC attrs: 0x1 [ 3.493284] rtc_cmos 00:05: setting system clock to 2026-09-11 12:17:59 UTC (1789129079) [ 3.500037] debug: unmapping init [mem 0xffffffff9a403000-0xffffffff9a5fffff] [ 3.503259] debug: unmapping init [mem 0xffffffff99182000-0xffffffff99458fff] [ 3.512118] Write protecting the kernel read-only data: 28672k [ 3.515383] debug: unmapping init [mem 0xffffffff97803000-0xffffffff979fffff] [ 3.518030] debug: unmapping init [mem 0xffffffff98114000-0xffffffff981fffff] [ 3.562353] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.573353] systemd[1]: Detected virtualization kvm. [ 3.575449] systemd[1]: Detected architecture x86-64. [ 3.577475] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.607683] systemd[1]: No hostname configured. [ 3.609525] systemd[1]: Set hostname to . [ 3.611502] random: systemd: uninitialized urandom read (16 bytes read) [ 3.614074] systemd[1]: Initializing machine ID from random generator. [ 3.758954] random: systemd: uninitialized urandom read (16 bytes read) [ 3.773625] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.779521] random: systemd: uninitialized urandom read (16 bytes read) [ 3.782535] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.787144] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Sockets. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.437278] device-mapper: uevent: version 1.0.3 [ 4.441109] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook.[ 4.866664] random: fast init done Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.274767] virtio_net virtio0 ens2: renamed from eth0 [ 5.359209] scsi host0: ata_piix [ 5.372224] scsi host1: ata_piix [ 5.374183] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.378300] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.242505] dracut-initqueue[579]: RTNETLINK answers: File exists [ 9.998451] random: crng init done [ 9.999828] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.577312] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.767361] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.043065] SELinux: Disabled at runtime. [ 12.102538] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.112408] systemd[1]: Detected virtualization kvm. [ 12.114484] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.634477] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.638553] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.643368] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.648182] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.652362] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.663626] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.670659] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ 12.902119] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 13.186640] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.511873] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.542632] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.713097] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.730421] EDAC sbridge: Ver: 1.1.2 [ 15.440588] Key type dns_resolver registered [ 15.812406] NFS: Registering the id_resolver key type [ 15.815349] Key type id_resolver registered [ 15.817668] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg622-server login: [ 77.486253] libcfs: loading out-of-tree module taints kernel. [ 77.557613] Key type ._llcrypt registered [ 77.559271] Key type .llcrypt registered [ 77.680533] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [ 95.371700] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 97.144776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 97.157536] alg: No test for adler32 (adler32-zlib) [ 98.727861] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 99.812766] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 101.655161] Key type lgssc registered [ 103.353630] Lustre: Echo OBD driver; http://www.lustre.org/ [ 121.863927] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 135.737299] hrtimer: interrupt took 9077590 ns [ 166.473471] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 181.855165] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 181.946743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 183.204791] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 183.238312] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 183.354798] Lustre: lustre-MDT0000: new disk, initializing [ 183.446259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 183.471487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 189.245938] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 203.763958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 203.908805] Lustre: 6510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 203.960241] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 203.969756] Lustre: Skipped 1 previous similar message [ 204.072311] Lustre: lustre-MDT0001: new disk, initializing [ 204.177561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 204.207237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 204.225365] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 209.273113] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 214.248885] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 224.010519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 224.244517] Lustre: lustre-OST0000: new disk, initializing [ 224.251921] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 224.256535] Lustre: 8448:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 224.358420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 228.452333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 228.466412] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 228.599387] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 230.856553] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 247.147393] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 247.354160] Lustre: lustre-OST0001: new disk, initializing [ 247.366649] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 247.382597] Lustre: 9521:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 247.471741] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 254.917417] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 255.021361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 255.047808] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 255.160764] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 266.153963] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 277.382986] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 283.416350] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing check_logdir /tmp/testlogs/ [ 288.246213] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing yml_node [ 292.478749] Lustre: DEBUG MARKER: Client: 2.17.57.133 [ 294.883830] Lustre: DEBUG MARKER: MDS: 2.17.57.133 [ 297.241725] Lustre: DEBUG MARKER: OSS: 2.17.57.133 [ 298.734982] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Sep 11 08:22:53 EDT 2026 [ 316.508274] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 317.885564] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 337.376971] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 337.382022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 337.390634] Lustre: Skipped 1 previous similar message [ 337.406805] Lustre: Skipped 3 previous similar messages [ 338.964672] Lustre: server umount lustre-MDT0000 complete [ 342.691552] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789129418 with bad export cookie 429006561981970051 [ 342.692973] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 342.696450] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 347.616673] LustreError: 6514:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 347.620120] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 347.628140] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 347.640309] LustreError: 6514:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 347.675684] Lustre: Skipped 3 previous similar messages [ 349.032635] Lustre: server umount lustre-MDT0001 complete [ 352.721892] Lustre: server umount lustre-OST0000 complete [ 356.895864] Lustre: server umount lustre-OST0001 complete [ 363.492839] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [ 373.165209] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 405.626193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 405.922817] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 405.951924] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 406.029162] Lustre: lustre-MDT0000: new disk, initializing [ 406.111518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 406.130446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 410.505058] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 420.786549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 420.909284] Lustre: 15072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 420.950229] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 420.954563] Lustre: Skipped 1 previous similar message [ 421.028959] Lustre: lustre-MDT0001: new disk, initializing [ 421.141512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 421.192690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 421.200042] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 425.847093] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 435.661545] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 441.666491] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 448.664979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 449.018851] Lustre: lustre-OST0000: new disk, initializing [ 449.027120] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 449.035561] Lustre: 17041:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 449.120981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 450.785293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 450.804577] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 450.923255] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 455.104604] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 463.882920] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 468.624656] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 468.842458] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 473.406622] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 473.666968] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 476.661517] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 476.672910] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 476.677332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 476.677341] Lustre: Skipped 1 previous similar message [ 476.695541] Lustre: Skipped 1 previous similar message [ 481.762186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 481.768733] Lustre: Skipped 1 previous similar message [ 482.005088] Lustre: server umount lustre-OST0000 complete [ 486.884075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 486.910749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 486.924074] Lustre: Skipped 2 previous similar messages [ 497.122529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 497.128774] Lustre: Skipped 3 previous similar messages [ 500.192160] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 500.542182] Lustre: server umount lustre-MDT0000 complete [ 502.241125] LustreError: 15083:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 502.263206] LustreError: 15083:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 504.302703] LustreError: 15064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789129580 with bad export cookie 429006561981971577 [ 504.306688] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 504.320428] LustreError: 15064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 504.810671] Lustre: server umount lustre-MDT0001 complete [ 509.513325] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 08:26:24 (1789129584) [ 511.799450] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 513.441607] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 08:26:28 (1789129588) [ 520.252112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 520.703174] LustreError: 19450:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 520.772256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 525.419082] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 525.803822] LustreError: 19451:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 530.914338] LustreError: 19450:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 533.354370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 537.925018] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 545.653477] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 551.896126] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 558.938249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 566.906777] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 577.924019] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 589.283059] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 589.287314] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 589.318455] Lustre: Skipped 1 previous similar message [ 589.329827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 589.345119] LustreError: Skipped 1 previous similar message [ 589.390817] Lustre: Skipped 1 previous similar message [ 594.418132] LustreError: 21273:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 594.442488] LustreError: 21273:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 594.566495] Lustre: server umount lustre-OST0000 complete [ 599.536674] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 599.575030] Lustre: Skipped 2 previous similar messages [ 613.343125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 613.604123] Lustre: server umount lustre-MDT0000 complete [ 614.880939] LustreError: 20708:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 614.907672] LustreError: 20708:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 617.656148] LustreError: 19431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789129693 with bad export cookie 429006561981972466 [ 617.670670] LustreError: 19431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 617.671664] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 618.295536] Lustre: server umount lustre-MDT0001 complete [ 630.632440] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 633.339075] Key type lgssc unregistered [ 633.740070] LNet: 22990:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 633.752248] LNetError: 22990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 633.771892] LNet: Removed LNI 192.168.206.122@tcp [ 634.905150] Key type .llcrypt unregistered [ 634.908095] Key type ._llcrypt unregistered [ 659.935248] Key type ._llcrypt registered [ 659.941038] Key type .llcrypt registered [ 660.031937] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 661.381640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 661.478446] alg: No test for adler32 (adler32-zlib) [ 662.583318] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 662.894653] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 664.639284] Key type lgssc registered [ 665.836437] Lustre: Echo OBD driver; http://www.lustre.org/ [ 673.466260] loop: module loaded [ 673.569779] loop0: detected capacity change from 0 to 204800000 [ 673.775167] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 673.783271] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 674.074296] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 679.934888] loop0: detected capacity change from 0 to 204800000 [ 680.119358] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 680.137446] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 680.157573] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 685.654500] loop0: detected capacity change from 0 to 204800000 [ 691.460348] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 691.506221] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 692.802812] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 692.858864] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 692.943807] Lustre: lustre-MDT0000: new disk, initializing [ 693.014382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 693.039639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 697.590632] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 706.626118] loop1: detected capacity change from 0 to 204800000 [ 712.430772] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 712.776386] Lustre: lustre-OST07c6: new disk, initializing [ 712.782885] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 712.785762] Lustre: Skipped 1 previous similar message [ 712.789437] Lustre: 26182:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 712.842230] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 718.697693] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 721.501204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 721.509673] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 721.554329] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 723.016028] Lustre: 25168:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 723.025340] Lustre: 25168:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 723.029488] Lustre: 25168:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 723.034541] Lustre: 25168:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 723.039674] Lustre: 25168:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 723.044117] Lustre: 25168:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 737.840169] Lustre: 25168:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 737.849634] Lustre: 25168:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 737.855368] Lustre: 25168:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 737.874321] Lustre: 25168:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 737.883777] Lustre: 25168:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 737.897409] Lustre: 25168:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 742.897916] Lustre: 25169:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 742.903829] Lustre: 25169:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message [ 742.909976] Lustre: 25169:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 742.914372] Lustre: 25169:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 742.918964] Lustre: 25169:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 742.927113] Lustre: 25169:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 742.931982] Lustre: 25169:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 742.936016] Lustre: 25169:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 742.940066] Lustre: 25169:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 742.943764] Lustre: 25169:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 742.947475] Lustre: 25169:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 742.951384] Lustre: 25169:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 746.153304] Lustre: 25168:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 746.167946] Lustre: 25168:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message [ 746.174034] Lustre: 25168:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 746.185373] Lustre: 25168:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 746.198177] Lustre: 25168:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 746.206603] Lustre: 25168:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 746.212492] Lustre: 25168:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 746.225436] Lustre: 25168:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 746.230107] Lustre: 25168:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 746.241374] Lustre: 25168:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 746.246657] Lustre: 25168:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 746.257661] Lustre: 25168:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 750.833970] Lustre: 25168:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 750.845510] Lustre: 25168:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message [ 750.854903] Lustre: 25168:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 750.865235] Lustre: 25168:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 750.874104] Lustre: 25168:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 750.882531] Lustre: 25168:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 750.900182] Lustre: 25168:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 750.914134] Lustre: 25168:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 750.923952] Lustre: 25168:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 750.936626] Lustre: 25168:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 750.947284] Lustre: 25168:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 750.959202] Lustre: 25168:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.621787] Lustre: 25169:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 762.630277] Lustre: 25169:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 5 previous similar messages [ 762.643699] Lustre: 25169:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 762.661234] Lustre: 25169:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 762.674121] Lustre: 25169:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 762.691118] Lustre: 25169:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 762.714300] Lustre: 25169:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 762.721891] Lustre: 25169:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 762.728155] Lustre: 25169:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 762.736037] Lustre: 25169:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 762.749037] Lustre: 25169:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 762.757178] Lustre: 25169:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 813.536931] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 813.558362] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 813.591910] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 817.859394] Lustre: server umount lustre-OST07c6 complete [ 827.806790] Lustre: server umount lustre-MDT0000 complete [ 846.010426] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 849.778648] Key type lgssc unregistered [ 850.159376] LNet: 28739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 850.167107] LNetError: 28739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 850.199413] LNet: Removed LNI 192.168.206.122@tcp [ 851.169554] Key type .llcrypt unregistered [ 851.171570] Key type ._llcrypt unregistered [ 872.421351] Key type ._llcrypt registered [ 872.431454] Key type .llcrypt registered [ 872.590949] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 873.756354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 873.829321] alg: No test for adler32 (adler32-zlib) [ 875.232683] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 875.543552] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 877.199218] Key type lgssc registered [ 879.048717] Lustre: Echo OBD driver; http://www.lustre.org/ [ 888.684414] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 888.713130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 890.641393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 896.879653] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 900.073074] LustreError: 30019:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 900.091342] LustreError: 30019:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 905.186668] LustreError: 30020:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 908.138129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 908.679622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 913.529291] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 921.259495] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 927.236700] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 934.354800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 934.886484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 935.930311] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 940.908898] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 949.633212] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 969.697458] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 969.715416] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 969.735203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 969.776622] Lustre: Skipped 1 previous similar message [ 973.894400] Lustre: server umount lustre-OST0000 complete [ 979.938036] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 979.944612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 979.959161] Lustre: Skipped 2 previous similar messages [ 979.988813] Lustre: Skipped 1 previous similar message [ 985.057636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 985.080961] Lustre: Skipped 1 previous similar message [ 985.887908] Lustre: server umount lustre-MDT0000 complete [ 989.158380] LustreError: 30750:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 989.190030] LustreError: 30750:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 990.087239] LustreError: 29999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789130066 with bad export cookie 17047589274386267905 [ 990.094161] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 990.109117] LustreError: 29999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 990.569394] Lustre: server umount lustre-MDT0001 complete [ 1003.461855] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 1007.135364] Key type lgssc unregistered [ 1007.461672] LNet: 33764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1007.472720] LNetError: 33764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1007.487888] LNet: Removed LNI 192.168.206.122@tcp [ 1008.273827] Key type .llcrypt unregistered [ 1008.276558] Key type ._llcrypt unregistered [ 1028.615909] Key type ._llcrypt registered [ 1028.620451] Key type .llcrypt registered [ 1028.691227] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 1029.628129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1029.652126] alg: No test for adler32 (adler32-zlib) [ 1031.007962] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 1031.433552] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 1033.175208] Key type lgssc registered [ 1034.545370] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1043.364364] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1043.396869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1044.888586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1049.698790] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 1054.181131] LustreError: 35039:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1054.195472] LustreError: 35039:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1058.411520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1058.839090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1062.992242] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 1073.701515] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1080.930605] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1089.166904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1089.441764] LustreError: 36860:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1089.719595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1095.172909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1097.206450] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 1106.147401] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1118.693822] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1118.694549] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1118.734522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1118.747120] Lustre: Skipped 1 previous similar message [ 1123.069626] Lustre: server umount lustre-OST0000 complete [ 1128.931204] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1128.953187] Lustre: Skipped 1 previous similar message [ 1128.966203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1128.975960] Lustre: Skipped 1 previous similar message [ 1134.057369] LustreError: 36241:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1134.114917] Lustre: server umount lustre-MDT0000 complete [ 1138.149615] LustreError: 35038:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1138.197020] LustreError: 35038:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1138.984165] LustreError: 35019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789130214 with bad export cookie 3652412500322137105 [ 1139.005202] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1139.615804] Lustre: server umount lustre-MDT0001 complete [ 1152.320388] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 1155.717902] Key type lgssc unregistered [ 1156.265038] LNet: 38577:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1156.272249] LNetError: 38577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1156.329951] LNet: Removed LNI 192.168.206.122@tcp [ 1157.785498] Key type .llcrypt unregistered [ 1157.787389] Key type ._llcrypt unregistered [ 1179.855392] Key type ._llcrypt registered [ 1179.857244] Key type .llcrypt registered [ 1179.970799] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 1180.954737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1181.030110] alg: No test for adler32 (adler32-zlib) [ 1182.042462] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 1182.266196] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 1183.911227] Key type lgssc registered [ 1185.095158] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1193.219395] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1193.239659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1194.760470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1199.179304] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 1204.212853] LustreError: 39977:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1204.224635] LustreError: 39977:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1208.320085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1208.865181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1213.801886] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 1221.897797] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1228.070892] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1236.047028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1236.418762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1239.497555] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1242.118198] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 1249.167970] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1258.981974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1259.002420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1259.007070] Lustre: Skipped 1 previous similar message [ 1259.045932] Lustre: Skipped 2 previous similar messages [ 1262.165749] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.22@tcp (stopping) [ 1264.105090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1264.120367] Lustre: Skipped 2 previous similar messages [ 1265.368069] Lustre: server umount lustre-MDT0000 complete [ 1267.266433] LustreError: 39974:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1269.158908] LustreError: 39957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789130345 with bad export cookie 14596569076893980799 [ 1269.169674] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1269.178500] LustreError: 39957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1269.216614] LustreError: 39973:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1269.218740] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1269.254907] Lustre: Skipped 1 previous similar message [ 1269.268184] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1272.387657] LustreError: 39973:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1274.336783] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1274.345453] Lustre: Skipped 1 previous similar message [ 1275.706982] Lustre: server umount lustre-MDT0001 complete [ 1290.345882] Lustre: server umount lustre-OST0000 complete [ 1309.039167] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 1311.795462] Key type lgssc unregistered [ 1312.219387] LNet: 43840:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1312.234329] LNetError: 43840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1312.265074] LNet: Removed LNI 192.168.206.122@tcp [ 1313.624088] Key type .llcrypt unregistered [ 1313.628168] Key type ._llcrypt unregistered [ 1338.183966] Key type ._llcrypt registered [ 1338.186571] Key type .llcrypt registered [ 1338.306311] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 1340.138703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1340.159243] alg: No test for adler32 (adler32-zlib) [ 1341.518585] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 1341.835383] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 1343.551140] Key type lgssc registered [ 1345.160082] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1354.319720] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1354.329771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1355.779265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1360.504543] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 1365.994203] LustreError: 45258:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1366.018385] LustreError: 45258:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1371.105144] LustreError: 45259:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1371.338026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1371.712737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1376.945713] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 1385.404946] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1392.810291] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1400.923695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1401.400630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1409.356228] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 1416.789666] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1425.981136] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1432.032547] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1432.038872] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1432.051521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1432.676512] Lustre: lustre-OST0000: Not available for connect from 192.168.206.22@tcp (stopping) [ 1436.132222] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1436.135458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1436.166127] Lustre: Skipped 1 previous similar message [ 1437.773310] LustreError: 47082:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1437.883226] Lustre: server umount lustre-OST0000 complete [ 1456.610377] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1456.614264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1456.648550] Lustre: Skipped 1 previous similar message [ 1456.656171] Lustre: Skipped 1 previous similar message [ 1461.745408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1461.757322] Lustre: Skipped 1 previous similar message [ 1462.034210] Lustre: server umount lustre-MDT0000 complete [ 1462.753186] LustreError: 45258:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1462.781888] LustreError: 45258:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1465.979405] LustreError: 45240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789130541 with bad export cookie 4932608858470445712 [ 1465.982993] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1465.995734] LustreError: 45240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1472.504260] Lustre: server umount lustre-MDT0001 complete [ 1483.856729] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 1486.779632] Key type lgssc unregistered [ 1487.133919] LNet: 48994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1487.153218] LNetError: 48994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1487.193234] LNet: Removed LNI 192.168.206.122@tcp [ 1488.031144] Key type .llcrypt unregistered [ 1488.037567] Key type ._llcrypt unregistered [ 1511.261412] Key type ._llcrypt registered [ 1511.265778] Key type .llcrypt registered [ 1511.383913] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 1512.589272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1512.628766] alg: No test for adler32 (adler32-zlib) [ 1513.714844] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 1514.022114] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 1515.695214] Key type lgssc registered [ 1516.876041] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1524.991833] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1525.008864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1526.470720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1531.625641] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 1536.994974] LustreError: 50413:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1537.030193] LustreError: 50413:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1540.723818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1541.105962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1546.197524] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 1554.187763] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1561.232753] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1568.803507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1569.201803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1572.284417] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1576.734234] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 1584.543810] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1590.792948] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1594.345564] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1606.662768] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-11 8h44m40s [ 1608.158851] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1612.271202] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1612.288481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1612.290705] Lustre: Skipped 1 previous similar message [ 1612.302127] Lustre: Skipped 2 previous similar messages [ 1612.881204] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.22@tcp (stopping) [ 1613.945990] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.22@tcp (stopping) [ 1613.964362] Lustre: Skipped 3 previous similar messages [ 1616.994107] Lustre: server umount lustre-MDT0000 complete [ 1619.027398] LustreError: 51617:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1621.192671] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1625.145781] LustreError: 50408:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1625.161298] LustreError: 50408:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 1628.808795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1628.930170] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1629.279568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1629.348181] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1630.314446] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1634.006200] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 1637.435879] Lustre: DEBUG MARKER: Wait for df (36714) ... [ 1638.960702] Lustre: DEBUG MARKER: done [ 1644.513803] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1644.524052] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1644.540031] Lustre: Skipped 1 previous similar message [ 1644.551725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1646.049055] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1649.428959] Lustre: server umount lustre-OST0000 complete [ 1656.290710] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1656.298311] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1656.317335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1656.337936] Lustre: Skipped 3 previous similar messages [ 1659.598867] Lustre: server umount lustre-MDT0000 complete [ 1663.457866] LustreError: 50414:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1663.477026] LustreError: 50414:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 1663.718397] LustreError: 50394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789130739 with bad export cookie 16479121585017361524 [ 1663.729163] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1663.733554] LustreError: 50394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1664.104602] Lustre: server umount lustre-MDT0001 complete [ 1674.245656] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 1677.395569] Key type lgssc unregistered [ 1677.710896] LNet: 55340:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1677.720680] LNetError: 55340:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1677.741317] LNet: Removed LNI 192.168.206.122@tcp [ 1678.730053] Key type .llcrypt unregistered [ 1678.738532] Key type ._llcrypt unregistered [ 1717.280937] Key type ._llcrypt registered [ 1717.283844] Key type .llcrypt registered [ 1717.379994] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 1718.438885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1718.468859] alg: No test for adler32 (adler32-zlib) [ 1719.566414] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 1719.849162] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 1721.584140] Key type lgssc registered [ 1722.644409] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1730.972593] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1730.991823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1732.365803] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1732.386918] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1732.560639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1737.594097] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 1747.152903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1747.330947] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1747.430620] Lustre: 57551:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1747.777432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1753.236777] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 1760.597648] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1766.141499] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1774.183721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1774.421556] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1774.629271] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1781.560898] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 1783.439748] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1789.902614] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1796.994992] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-11 8h47m51s [ 1798.495481] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1804.544685] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1806.665296] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1806.707691] LustreError: 57559:0:(mdt_handler.c:2352:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1826.895490] Lustre: lustre-MDT0000: Client 89e67a9c-fe38-48ef-9faf-376613668fc8 (at 192.168.206.22@tcp) reconnecting [ 1846.808055] LustreError: 57559:0:(mdt_handler.c:2352:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1849.616922] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1852.121380] Lustre: DEBUG MARKER: done [ 1860.064562] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1860.074299] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1860.194668] Lustre: server umount lustre-OST0000 complete [ 1865.697272] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1865.698761] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1865.708851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1865.734517] Lustre: Skipped 1 previous similar message [ 1866.784763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1866.798745] Lustre: Skipped 2 previous similar messages [ 1871.842050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1871.851161] Lustre: Skipped 1 previous similar message [ 1874.931132] LustreError: 61130:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (000000006fb0a067) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 1876.994691] Lustre: server umount lustre-MDT0000 complete [ 1877.986852] LustreError: 60634:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1878.011418] LustreError: 60634:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 1880.828711] LustreError: 57543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789130956 with bad export cookie 16607553034549814252 [ 1880.832457] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1880.846251] LustreError: 57543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1881.339725] Lustre: server umount lustre-MDT0001 complete [ 1891.491184] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 1894.171515] Key type lgssc unregistered [ 1894.492666] LNet: 61887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1894.513628] LNetError: 61887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1894.526980] LNet: Removed LNI 192.168.206.122@tcp [ 1895.376621] Key type .llcrypt unregistered [ 1895.379467] Key type ._llcrypt unregistered [ 1939.927192] Key type ._llcrypt registered [ 1939.929731] Key type .llcrypt registered [ 1940.017394] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 1941.378287] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1941.393360] alg: No test for adler32 (adler32-zlib) [ 1942.530823] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 1942.825030] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 1944.544836] Key type lgssc registered [ 1945.813963] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1949.275631] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1949.321483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1950.638920] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1950.685945] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1950.898851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1951.715445] Lustre: Failing over lustre-MDT0000 [ 1951.945450] Lustre: server umount lustre-MDT0000 complete [ 1958.677372] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 08:50:33 (1789131033) [ 1965.229670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1965.531447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1970.520206] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 1980.049677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1980.174984] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1980.218569] Lustre: 64684:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1980.430547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1985.289705] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 1993.176546] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1999.510167] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2007.845232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2008.164589] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2008.591180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2015.286307] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 2016.877906] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2021.924840] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2028.028637] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2034.301097] Lustre: server umount lustre-MDT0000 complete [ 2036.707254] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2036.722288] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2037.731835] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2038.074253] LustreError: 64676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131114 with bad export cookie 3514455400907699235 [ 2038.082124] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2038.093799] LustreError: 64676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2038.454658] Lustre: server umount lustre-MDT0001 complete [ 2041.929205] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2045.894488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2053.399106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2058.208139] Lustre: 63609:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789131118/real 1789131118] req@0000000091b20b79 x1876039846369152/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789131134 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2058.237297] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2058.247128] Lustre: Skipped 1 previous similar message [ 2059.234303] Lustre: 63610:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789131119/real 1789131119] req@00000000e69b6b98 x1876039846369280/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789131135 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2063.327229] Lustre: 63608:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789131123/real 1789131123] req@000000009dde3f3b x1876039846369408/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789131139 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2064.419225] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x30c5da91160e7d0d [ 2064.429450] Lustre: MGC192.168.206.122@tcp: Connection restored to 0@lo (at 0@lo) [ 2064.654065] LustreError: 66535:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2064.667927] LustreError: 66535:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 2064.732133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2064.826819] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2068.796703] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 2068.833708] LustreError: 66536:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2068.848089] LustreError: 66536:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2070.418118] Lustre: 63610:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789131130/real 1789131130] req@00000000561a5d31 x1876039846369792/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789131146 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2070.451836] Lustre: 63610:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2074.084197] LustreError: 68456:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2074.109088] LustreError: 68456:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2076.567396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2076.942412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2077.995306] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2081.400919] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 2088.710219] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2094.648418] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2102.756378] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2102.764721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2102.790757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2103.779767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2103.792241] Lustre: Skipped 2 previous similar messages [ 2108.479226] Lustre: server umount lustre-MDT0000 complete [ 2108.907107] LustreError: 69547:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2108.928044] LustreError: 69547:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2112.453866] LustreError: 65419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131188 with bad export cookie 3514455400907701517 [ 2112.464603] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2112.477815] LustreError: 65419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2112.949104] Lustre: server umount lustre-MDT0001 complete [ 2119.696305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2129.470416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2130.399141] Lustre: 63609:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789131190/real 1789131190] req@00000000d5e021bf x1876039846396544/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789131206 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2130.426507] Lustre: 63609:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2130.431560] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2130.440316] Lustre: Skipped 2 previous similar messages [ 2137.912363] LustreError: 66535:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2138.005313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2138.078200] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2138.977229] Lustre: 63607:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789131198/real 1789131198] req@00000000fc35bd91 x1876039846397056/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789131214 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2139.010233] Lustre: 63607:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 2142.485990] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 2146.274667] LustreError: 66536:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2146.291048] LustreError: 66536:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 2151.362442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2155.930049] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 2157.042929] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2157.052422] Lustre: Skipped 1 previous similar message [ 2159.153572] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2163.545040] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2169.713743] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2177.511346] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2177.521321] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2177.543276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2179.555387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2179.572783] Lustre: Skipped 2 previous similar messages [ 2182.852059] Lustre: server umount lustre-MDT0000 complete [ 2184.675312] LustreError: 66536:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2184.701452] LustreError: 66536:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 2186.749404] LustreError: 64675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131262 with bad export cookie 3514455400907702896 [ 2186.752664] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2186.759876] LustreError: 64675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2187.186671] Lustre: server umount lustre-MDT0001 complete [ 2192.983232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2196.205690] Lustre: DEBUG MARKER: files compared the same [ 2208.679583] Lustre: server umount lustre-OST0000 complete [ 2225.720057] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 2229.331679] Key type lgssc unregistered [ 2229.632817] LNet: 74419:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2229.649139] LNetError: 74419:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2229.667470] LNet: Removed LNI 192.168.206.122@tcp [ 2230.456086] Key type .llcrypt unregistered [ 2230.457696] Key type ._llcrypt unregistered [ 2254.575893] Key type ._llcrypt registered [ 2254.579852] Key type .llcrypt registered [ 2254.677437] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 2255.606482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2255.639493] alg: No test for adler32 (adler32-zlib) [ 2256.662863] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 2256.841778] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 2258.487325] Key type lgssc registered [ 2259.505513] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2266.655647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2266.678246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2268.248779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2272.649266] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug +malloc all [ 2278.368352] LustreError: 75836:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2278.381703] LustreError: 75836:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2281.297120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2281.606150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2285.683503] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug +malloc all [ 2293.613899] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2298.997020] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2305.487345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2305.820228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2312.135961] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug +malloc all [ 2314.027254] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2319.612292] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2327.519787] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2327.532712] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2327.545721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2329.093390] Lustre: server umount lustre-OST0000 complete [ 2333.152049] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2333.164458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2333.189544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2334.687933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2334.689507] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2339.809250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2339.817033] Lustre: Skipped 2 previous similar messages [ 2342.955944] Lustre: server umount lustre-MDT0000 complete [ 2344.931832] LustreError: 76890:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2344.950153] LustreError: 76890:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2347.132478] LustreError: 75818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131423 with bad export cookie 14076154643310452547 [ 2347.162458] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2347.680803] Lustre: server umount lustre-MDT0001 complete [ 2359.562739] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 2362.089186] Key type lgssc unregistered [ 2362.400587] LNet: 79373:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2362.413868] LNetError: 79373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2363.434145] LNet: Removed LNI 192.168.206.122@tcp [ 2364.240258] Key type .llcrypt unregistered [ 2364.245725] Key type ._llcrypt unregistered [ 2385.728533] Key type ._llcrypt registered [ 2385.734654] Key type .llcrypt registered [ 2385.858222] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 2386.864497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2386.928800] alg: No test for adler32 (adler32-zlib) [ 2387.958317] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 2388.173526] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 2389.895165] Key type lgssc registered [ 2391.000730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2397.764753] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2397.778831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2413.411490] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2418.527396] LustreError: 80758:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.122@tcp: failed processing log, type 4: rc = -110 [ 2445.151677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2451.057695] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 2459.023988] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2466.506923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2466.734938] Lustre: *** cfs_fail_loc=706, val=0*** [ 2467.103557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2467.186365] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2471.358482] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 2472.438759] LustreError: 80783:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2472.454351] LustreError: 80783:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2477.542135] LustreError: 80784:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2479.159748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2479.488974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2483.290916] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 2490.268931] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2495.878842] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2498.016423] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2498.023488] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2498.040985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2500.064996] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2500.078815] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2500.088380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2501.093969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2501.100197] Lustre: Skipped 1 previous similar message [ 2506.230191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2506.244761] Lustre: Skipped 1 previous similar message [ 2511.335363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2511.341184] Lustre: Skipped 1 previous similar message [ 2512.351152] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2512.476722] Lustre: server umount lustre-OST0000 complete [ 2516.448820] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2521.570106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2521.581884] Lustre: Skipped 3 previous similar messages [ 2522.770109] Lustre: server umount lustre-MDT0000 complete [ 2526.560952] LustreError: 80764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131602 with bad export cookie 11012561266198432207 [ 2526.562727] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2526.577149] LustreError: 80764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2526.858277] Lustre: server umount lustre-MDT0001 complete [ 2537.301120] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 2540.030345] Key type lgssc unregistered [ 2540.297278] LNet: 84379:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2540.305113] LNetError: 84379:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2540.319507] LNet: Removed LNI 192.168.206.122@tcp [ 2541.114293] Key type .llcrypt unregistered [ 2541.116306] Key type ._llcrypt unregistered [ 2564.664090] Key type ._llcrypt registered [ 2564.669408] Key type .llcrypt registered [ 2564.813834] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 2565.768664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2565.830410] alg: No test for adler32 (adler32-zlib) [ 2566.894855] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 2567.090677] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 2568.823190] Key type lgssc registered [ 2569.898380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2577.050893] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2577.099900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2582.542805] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 2589.767654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2590.411330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2594.876559] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 2595.812052] LustreError: 86494:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2595.836947] LustreError: 86494:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2600.932907] LustreError: 86497:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2600.947321] LustreError: 86497:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2602.922907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2603.275941] LustreError: 86494:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2603.292097] LustreError: 86494:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2603.338159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2608.539443] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 2608.609781] LustreError: 86498:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2608.630091] LustreError: 86498:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2613.737969] LustreError: 86497:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2613.758168] LustreError: 86497:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2616.626254] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2616.637824] Lustre: MGS: Client d12babcc-8106-497b-a748-139888c31d61 (at 0@lo) reconnecting [ 2616.650942] LustreError: 86508:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2616.883560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2616.962384] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2620.918549] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 2637.287782] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2637.292821] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2637.301311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2637.312371] Lustre: Skipped 1 previous similar message [ 2637.318164] LustreError: Skipped 1 previous similar message [ 2639.234643] Lustre: server umount lustre-OST0000 complete [ 2647.520701] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2647.523822] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2647.550472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2647.550480] Lustre: Skipped 1 previous similar message [ 2652.641870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2652.646328] Lustre: Skipped 2 previous similar messages [ 2657.248095] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2657.522844] Lustre: server umount lustre-MDT0000 complete [ 2657.777072] LustreError: 86497:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2657.799715] LustreError: 86497:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 2662.767584] Lustre: server umount lustre-MDT0001 complete [ 2667.430634] Lustre: server umount MGS complete [ 2678.890598] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 09:02:33 (1789131753) [ 2709.983063] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [ 2718.763200] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 2754.961649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2759.899634] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 2770.857354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2771.319809] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2771.412897] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2771.511923] Lustre: lustre-MDT0001: new disk, initializing [ 2771.665742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2771.725716] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2771.747203] Lustre: 93665:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2771.765124] LustreError: 93665:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2777.670502] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 2789.964383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2790.341518] Lustre: lustre-OST0000: new disk, initializing [ 2790.345341] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2790.352854] Lustre: 94433:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2790.443184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2797.087352] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 2799.649208] Lustre: 94872:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000058630964 x1876040501732096/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789131891 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2806.451777] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2813.837848] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2813.849224] Lustre: MGS: Client b73b8fcb-dfa5-43ff-9e3f-6dbd6e5f95b9 (at 0@lo) reconnecting [ 2813.898629] Lustre: 92934:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2813.911988] LustreError: 93650:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2813.923533] LustreError: 93650:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2813.941606] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2814.024151] Lustre: lustre-MDT0000: new disk, initializing [ 2814.171543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2814.199456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2814.223410] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2815.274949] Lustre: 95501:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000c2356939 x1876040501738880/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1789131907 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2818.597527] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 2820.385928] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2824.369758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2824.380767] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2824.449386] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2847.714713] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2847.727558] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2847.739805] Lustre: Skipped 1 previous similar message [ 2847.747918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2847.757447] Lustre: Skipped 1 previous similar message [ 2852.840985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2853.170579] Lustre: server umount lustre-OST0000 complete [ 2857.954283] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2857.962160] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2857.964376] Lustre: Skipped 2 previous similar messages [ 2863.074477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2863.087253] Lustre: Skipped 4 previous similar messages [ 2871.263190] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2871.488482] Lustre: server umount lustre-MDT0000 complete [ 2873.323973] LustreError: 93634:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2873.338909] LustreError: 93634:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2881.461888] Lustre: server umount lustre-MDT0001 complete [ 2885.417278] Lustre: server umount MGS complete [ 2897.511102] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 09:06:11 (1789131971) [ 2916.702334] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 2919.018568] Key type lgssc unregistered [ 2919.406360] LNet: 98006:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2919.415476] LNetError: 98006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2919.445035] LNet: Removed LNI 192.168.206.122@tcp [ 2920.368711] Key type .llcrypt unregistered [ 2920.371066] Key type ._llcrypt unregistered [ 2936.484632] Key type ._llcrypt registered [ 2936.487741] Key type .llcrypt registered [ 2936.590068] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 2937.820528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2937.958514] alg: No test for adler32 (adler32-zlib) [ 2939.005698] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 2939.290253] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 2940.991226] Key type lgssc registered [ 2942.176714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2949.210646] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2949.234331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2950.812266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2955.257200] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 2960.364735] LustreError: 99228:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2960.396403] LustreError: 99228:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2963.936151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2964.393849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2968.722404] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 2975.900391] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2981.848885] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2988.150877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2988.644794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2994.556889] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 2995.846198] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3001.974701] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3010.538637] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3010.547528] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3010.556793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3011.556867] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3011.561732] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3011.562051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3011.580254] Lustre: Skipped 1 previous similar message [ 3015.576907] Lustre: server umount lustre-OST0000 complete [ 3020.771690] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3020.777641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3020.787655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3025.684886] Lustre: server umount lustre-MDT0000 complete [ 3026.916138] LustreError: 100663:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3026.948092] LustreError: 100663:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3029.090330] LustreError: 99209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789132105 with bad export cookie 7796488413058531590 [ 3029.093754] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3029.114840] LustreError: 99209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3029.418196] Lustre: server umount lustre-MDT0001 complete [ 3038.706292] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 3041.402206] Key type lgssc unregistered [ 3041.784083] LNet: 102765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3041.800629] LNetError: 102765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3041.822173] LNet: Removed LNI 192.168.206.122@tcp [ 3042.803343] Key type .llcrypt unregistered [ 3042.806962] Key type ._llcrypt unregistered [ 3056.948782] Key type ._llcrypt registered [ 3056.951946] Key type .llcrypt registered [ 3057.078863] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 3058.299954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3058.322536] alg: No test for adler32 (adler32-zlib) [ 3059.648919] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 3060.024739] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 3061.753098] Key type lgssc registered [ 3063.179572] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3070.287808] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3070.288079] LustreError: 103828:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3070.291689] Lustre: Skipped 1 previous similar message [ 3070.291727] LustreError: 103829:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3070.321860] LustreError: 103828:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 3070.325051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3070.328308] LustreError: 103828:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3070.328362] LustreError: 103828:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3072.091700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3080.527465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3081.008487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3086.001571] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 3093.164191] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing lsmod [ 3096.160288] LustreError: 105034:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3096.219677] LustreError: 105036:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3096.240865] LustreError: 105034:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4926 [ 3096.243518] LustreError: 105034:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3096.254960] LustreError: 105034:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3096.261160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3096.727993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3097.715800] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3101.664706] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3101.678627] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3101.690695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3103.206411] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3103.212025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3103.224576] Lustre: Skipped 1 previous similar message [ 3103.232418] Lustre: Skipped 1 previous similar message [ 3105.324369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3105.329352] Lustre: Skipped 3 previous similar messages [ 3106.239363] Lustre: server umount lustre-MDT0000 complete [ 3109.588054] LustreError: 105076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789132185 with bad export cookie 1407544670457810538 [ 3109.588929] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3109.596348] LustreError: 105076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3110.376476] LustreError: 105072:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3110.378122] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3110.397445] LustreError: 105072:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3110.422161] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3111.519344] LustreError: 104298:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000005e91a4a4 x1876041018660864/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3111.693276] Lustre: server umount lustre-MDT0001 complete [ 3114.938555] Lustre: server umount lustre-OST0000 complete [ 3123.222644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3123.811588] LustreError: 106098:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3123.901476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3127.795724] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 3129.315713] LustreError: 106099:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3133.409384] LustreError: 106098:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3135.533973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3135.829567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3139.940529] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 3147.019438] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3153.604639] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3160.784405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3161.135915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3167.196924] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 3168.319187] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3174.794570] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3188.704757] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3188.709220] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3188.730869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3193.470731] Lustre: server umount lustre-OST0000 complete [ 3197.409116] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3197.415505] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3197.430988] Lustre: Skipped 1 previous similar message [ 3197.435266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3197.437823] Lustre: Skipped 2 previous similar messages [ 3203.805742] Lustre: server umount lustre-MDT0000 complete [ 3206.118554] LustreError: 106099:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3206.143912] LustreError: 106099:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3208.225453] LustreError: 106080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789132284 with bad export cookie 1407544670457811462 [ 3208.231068] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3208.237111] LustreError: 106080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3208.862932] Lustre: server umount lustre-MDT0001 complete [ 3219.777909] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 3222.274226] Key type lgssc unregistered [ 3222.598690] LNet: 109653:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3222.603927] LNetError: 109653:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3222.618318] LNet: Removed LNI 192.168.206.122@tcp [ 3223.435331] Key type .llcrypt unregistered [ 3223.439752] Key type ._llcrypt unregistered [ 3247.457789] Key type ._llcrypt registered [ 3247.461716] Key type .llcrypt registered [ 3247.591653] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 3248.812884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3248.933723] alg: No test for adler32 (adler32-zlib) [ 3250.034733] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 3250.268405] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 3251.943178] Key type lgssc registered [ 3253.223484] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3261.385752] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3261.406535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3262.935779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3267.465867] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 3272.166317] LustreError: 111075:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3272.195208] LustreError: 111075:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3277.283204] LustreError: 111074:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3278.233390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3278.668188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3282.795174] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 3290.325171] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3296.637510] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3303.565882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3303.945426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3307.069327] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3309.760510] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 3318.137289] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3326.451058] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3330.045758] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3336.382942] Lustre: server umount lustre-OST0000 complete [ 3340.257103] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3340.267708] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3340.286712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3343.841332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3343.856500] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3348.961311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3348.972944] Lustre: Skipped 2 previous similar messages [ 3351.159662] Lustre: server umount lustre-MDT0000 complete [ 3354.017324] Lustre: 110384:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789132414/real 1789132414] req@000000002a35ce85 x1876041217903104/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789132430 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3354.048229] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3354.069694] LustreError: 111075:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3354.083378] LustreError: 111075:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3354.935932] LustreError: 111055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789132430 with bad export cookie 12999456481498962084 [ 3354.938548] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3354.953868] LustreError: 111055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3355.320929] Lustre: server umount lustre-MDT0001 complete [ 3365.890911] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 3368.380403] Key type lgssc unregistered [ 3368.691960] LNet: 114714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3368.706864] LNetError: 114714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3369.769543] LNet: Removed LNI 192.168.206.122@tcp [ 3370.623754] Key type .llcrypt unregistered [ 3370.627056] Key type ._llcrypt unregistered [ 3390.529318] Key type ._llcrypt registered [ 3390.538690] Key type .llcrypt registered [ 3390.650600] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 3391.938331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3392.027311] alg: No test for adler32 (adler32-zlib) [ 3393.012497] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 3393.202952] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 3394.855583] Key type lgssc registered [ 3395.614663] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3402.294768] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3402.332436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3403.775685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3408.742740] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 3413.992371] LustreError: 115938:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3414.017440] LustreError: 115938:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3417.278021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3417.686058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3421.829203] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 3428.938691] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3433.748276] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3440.090189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3440.292168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3447.837832] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 3448.515943] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3458.081208] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3468.258380] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3468.270817] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3468.304390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3469.287953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3469.290839] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3469.309587] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3473.669748] Lustre: server umount lustre-OST0000 complete [ 3479.008288] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3479.022694] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3479.040356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3479.049441] Lustre: Skipped 1 previous similar message [ 3484.645387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3484.664905] Lustre: Skipped 2 previous similar messages [ 3489.761439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3489.773789] Lustre: Skipped 1 previous similar message [ 3491.296104] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3491.594063] Lustre: server umount lustre-MDT0000 complete [ 3494.775866] LustreError: 115918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789132570 with bad export cookie 6746874561324999295 [ 3494.780852] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3494.786760] LustreError: 115918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3495.138839] Lustre: server umount lustre-MDT0001 complete [ 3505.362827] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 3507.564534] Key type lgssc unregistered [ 3507.869193] LNet: 119473:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3507.879263] LNetError: 119473:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3507.898566] LNet: Removed LNI 192.168.206.122@tcp [ 3508.532463] Key type .llcrypt unregistered [ 3508.534795] Key type ._llcrypt unregistered [ 3533.225569] Key type ._llcrypt registered [ 3533.227671] Key type .llcrypt registered [ 3533.353446] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 3534.192754] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3534.244085] alg: No test for adler32 (adler32-zlib) [ 3535.323608] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 3535.660943] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 3537.423286] Key type lgssc registered [ 3538.504188] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3545.289108] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3545.304556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3546.835609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3552.089754] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 3556.322584] LustreError: 120900:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3556.341206] LustreError: 120900:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3560.609520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3561.052179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3565.715201] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 3573.547624] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3579.369252] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3586.193331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3586.549217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3591.676087] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3592.790876] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 3600.078678] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3609.673790] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3612.937494] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3612.942757] Lustre: Skipped 2 previous similar messages [ 3616.810539] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3616.838784] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3616.923620] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3616.932880] Lustre: Skipped 2 previous similar messages [ 3623.040064] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3623.044217] Lustre: Skipped 1 previous similar message [ 3628.592130] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.22@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3628.603689] Lustre: Skipped 2 previous similar messages [ 3629.806467] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.22@tcp 192.168.0.[2,10]@tcp [ 3640.213974] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.22/24@tcp in log lustre-MDT0000 [ 3640.219441] Lustre: Skipped 2 previous similar messages [ 3645.509970] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.22/24@tcp [ 3645.523343] Lustre: Skipped 2 previous similar messages [ 3652.064392] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3652.073580] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3652.082570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3652.098509] Lustre: Skipped 1 previous similar message [ 3657.191150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3657.200450] Lustre: Skipped 2 previous similar messages [ 3657.783404] Lustre: server umount lustre-OST0000 complete [ 3662.304601] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3662.322193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3662.330397] Lustre: Skipped 1 previous similar message [ 3667.430100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3667.445488] Lustre: Skipped 1 previous similar message [ 3668.058219] Lustre: server umount lustre-MDT0000 complete [ 3672.205386] LustreError: 124290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789132748 with bad export cookie 11546891381303068126 [ 3672.208726] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3672.229538] LustreError: 124290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3672.690253] Lustre: server umount lustre-MDT0001 complete [ 3684.044639] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 3686.813289] Key type lgssc unregistered [ 3687.136637] LNet: 125803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3687.148684] LNetError: 125803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3688.169355] LNet: Removed LNI 192.168.206.122@tcp [ 3689.187379] Key type .llcrypt unregistered [ 3689.189495] Key type ._llcrypt unregistered [ 3719.352409] Key type ._llcrypt registered [ 3719.355058] Key type .llcrypt registered [ 3719.433839] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 3720.563048] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3720.635518] alg: No test for adler32 (adler32-zlib) [ 3721.726279] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 3722.096596] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 3723.831187] Key type lgssc registered [ 3725.050429] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3731.822623] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3731.832899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3733.198618] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3733.234492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3733.240863] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.22/24@tcp [ 3736.875807] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 3743.713941] LustreError: 127366:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3743.729981] LustreError: 127366:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3744.827955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3745.214947] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3745.244432] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.22/24@tcp [ 3745.299464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3750.704248] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 3758.471622] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3764.464928] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3771.411612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3771.741316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3777.821482] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 3778.940923] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3785.160042] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3799.520090] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3799.532590] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3799.549806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3800.549724] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3800.552254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3800.592786] Lustre: Skipped 1 previous similar message [ 3805.413940] Lustre: server umount lustre-OST0000 complete [ 3810.791360] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3810.801322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3810.814445] Lustre: Skipped 1 previous similar message [ 3810.833568] Lustre: Skipped 1 previous similar message [ 3815.907549] LustreError: 127366:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3816.028166] Lustre: server umount lustre-MDT0000 complete [ 3819.878997] LustreError: 127346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789132895 with bad export cookie 6195741870980369716 [ 3819.879930] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3819.886594] LustreError: 127346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3820.386714] Lustre: server umount lustre-MDT0001 complete [ 3831.095565] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 3833.680949] Key type lgssc unregistered [ 3833.976789] LNet: 130958:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3833.985795] LNetError: 130958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3834.004042] LNet: Removed LNI 192.168.206.122@tcp [ 3834.856963] Key type .llcrypt unregistered [ 3834.858694] Key type ._llcrypt unregistered [ 3860.751995] Key type ._llcrypt registered [ 3860.758315] Key type .llcrypt registered [ 3860.879812] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [ 3876.884671] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 3877.951482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3878.100788] alg: No test for adler32 (adler32-zlib) [ 3879.202988] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 3879.451969] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 3881.143342] Key type lgssc registered [ 3882.363879] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3917.382982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3917.398512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3918.692342] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3918.729953] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3918.854170] Lustre: lustre-MDT0000: new disk, initializing [ 3918.944826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3918.973836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3923.426808] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 3933.717451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3933.847324] Lustre: 134362:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3933.904905] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3933.910785] Lustre: Skipped 1 previous similar message [ 3933.989905] Lustre: lustre-MDT0001: new disk, initializing [ 3934.099108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3934.149965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3934.167914] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3939.402773] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 3949.957779] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3956.129548] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3962.876097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3963.281181] Lustre: lustre-OST0000: new disk, initializing [ 3963.303123] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3963.317723] Lustre: 136332:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3963.470929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3971.401958] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 3971.660506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3971.678335] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3971.849079] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3982.164710] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3987.920942] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3988.232942] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3993.859414] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3994.058528] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3997.667982] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3997.676502] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3997.686181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4002.482950] Lustre: server umount lustre-OST0000 complete [ 4007.905374] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4007.927503] Lustre: Skipped 1 previous similar message [ 4007.930791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4007.937168] Lustre: Skipped 2 previous similar messages [ 4013.024570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4013.030687] Lustre: Skipped 1 previous similar message [ 4013.210957] Lustre: server umount lustre-MDT0000 complete [ 4016.727282] LustreError: 134355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789133092 with bad export cookie 11619539287285828937 [ 4016.734465] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4017.075931] Lustre: server umount lustre-MDT0001 complete [ 4026.424614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4026.893485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4031.011575] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 4031.973759] LustreError: 138554:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4031.989087] LustreError: 138554:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4036.078527] LustreError: 138553:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4038.983437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4039.327246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4043.633564] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 4050.765437] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4057.186365] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4064.170420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4064.586939] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4071.010164] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 4077.898469] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4082.758505] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4082.935991] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4086.980333] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4087.177569] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4097.446387] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4097.752268] Lustre: lustre-OST0001: new disk, initializing [ 4097.760291] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4097.775539] Lustre: 141928:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4097.953164] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4104.610689] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 4105.851016] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4105.862809] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4105.956768] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4113.278582] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4113.542312] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4118.334370] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4118.566599] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4124.496196] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff898b82e7d800.ost_server_uuid 50 [ 4126.047329] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff898b82e7d800.ost_server_uuid in IDLE state after 0 sec [ 4136.928101] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4136.934357] LustreError: Skipped 1 previous similar message [ 4136.934446] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4136.947843] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4136.980264] Lustre: Skipped 2 previous similar messages [ 4140.033776] Lustre: server umount lustre-OST0001 complete [ 4142.059716] LustreError: 142215:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4142.074913] LustreError: 142215:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4145.632470] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4145.643956] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4145.654153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4145.659850] Lustre: Skipped 1 previous similar message [ 4147.172678] LustreError: 140373:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4147.191397] LustreError: 140373:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4149.864827] Lustre: server umount lustre-OST0000 complete [ 4157.408549] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4157.408707] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4157.414364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4157.425121] Lustre: Skipped 2 previous similar messages [ 4157.435992] Lustre: Skipped 2 previous similar messages [ 4159.969386] Lustre: server umount lustre-MDT0000 complete [ 4162.529269] LustreError: 138554:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4162.552401] LustreError: 138554:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4163.468372] LustreError: 141926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789133239 with bad export cookie 11619539287285829819 [ 4163.473899] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4163.494517] LustreError: 141926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4163.944413] Lustre: server umount lustre-MDT0001 complete [ 4182.528681] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 4186.085262] Key type lgssc unregistered [ 4186.394579] LNet: 144933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4186.406645] LNetError: 144933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4187.432819] LNet: Removed LNI 192.168.206.122@tcp [ 4188.264718] Key type .llcrypt unregistered [ 4188.270916] Key type ._llcrypt unregistered [ 4221.226548] Key type ._llcrypt registered [ 4221.228941] Key type .llcrypt registered [ 4221.392861] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [ 4235.363625] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 4236.202449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4236.325517] alg: No test for adler32 (adler32-zlib) [ 4237.410660] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 4237.737833] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 4239.400412] Key type lgssc registered [ 4240.386096] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4273.888219] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4273.948777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4275.307324] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4275.337955] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4275.443572] Lustre: lustre-MDT0000: new disk, initializing [ 4275.553761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4275.574761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4280.361524] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 4290.699742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4290.838517] Lustre: 149088:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4290.892749] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4290.895810] Lustre: Skipped 1 previous similar message [ 4290.950908] Lustre: lustre-MDT0001: new disk, initializing [ 4291.013649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4291.065118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4291.078625] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4296.164493] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 4305.643432] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4311.775799] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4319.566592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4319.867930] Lustre: lustre-OST0000: new disk, initializing [ 4319.872302] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4319.882717] Lustre: 151058:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4320.009440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4325.974861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4325.989125] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4326.102273] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4326.946180] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 4337.402975] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4342.155660] Lustre: Failing over lustre-OST0000 [ 4342.436137] Lustre: server umount lustre-OST0000 complete [ 4359.647227] Lustre: 147311:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789133419/real 1789133419] req@000000002561bba7 x1876042252847488/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1789133435 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4359.678405] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4362.209459] Lustre: 147312:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789133422/real 1789133422] req@00000000c256f7cf x1876042252848512/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789133438 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4362.443628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4362.784821] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4362.853601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4362.883427] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4363.872124] Lustre: 147311:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789133422/real 1789133422] req@0000000004f8740d x1876042252848384/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789133438 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4363.902373] Lustre: 147311:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4364.842674] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4368.863085] Lustre: 147313:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789133427/real 1789133427] req@00000000d22c8b01 x1876042252849792/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789133443 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4369.965793] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 4370.037997] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4370.038257] Lustre: lustre-OST0000: Recovery over after 0:06, of 3 clients 3 recovered and 0 were evicted. [ 4373.139195] Lustre: 147314:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789133433/real 1789133433] req@000000002761a025 x1876042252850816/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789133449 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4373.195937] Lustre: 147314:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4374.733863] Lustre: Failing over lustre-MDT0000 [ 4374.993985] Lustre: server umount lustre-MDT0000 complete [ 4375.020307] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4375.033614] Lustre: Skipped 1 previous similar message [ 4380.131283] LustreError: 149095:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4380.145115] LustreError: 149095:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 4385.251484] LustreError: 149099:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4385.268608] LustreError: 149099:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4390.371509] LustreError: 149093:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4390.396079] LustreError: 149093:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4391.391799] Lustre: 147312:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789133451/real 1789133451] req@0000000005f13172 x1876042252860928/t0(0) o400->MGC192.168.206.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789133467 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4391.428508] Lustre: 147312:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4391.436838] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4394.435797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4395.578573] LustreError: 149095:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4395.602859] LustreError: 149095:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 4401.634767] LustreError: 149095:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4401.666533] LustreError: 149095:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 4401.744516] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.22@tcp (not set up) [ 4402.029230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4402.090537] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4406.853799] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4407.290436] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4407.303319] Lustre: Skipped 1 previous similar message [ 4407.349447] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4407.450035] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4408.240640] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 4421.604195] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4421.621393] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4421.640416] Lustre: Skipped 1 previous similar message [ 4421.657849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4424.852679] Lustre: server umount lustre-OST0000 complete [ 4432.872067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4432.873552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4432.874182] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4432.892070] Lustre: Skipped 2 previous similar messages [ 4432.915814] Lustre: Skipped 3 previous similar messages [ 4435.860473] Lustre: server umount lustre-MDT0000 complete [ 4437.984884] LustreError: 153737:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4438.008130] LustreError: 153737:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4439.567332] LustreError: 149078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789133515 with bad export cookie 13831657460200819486 [ 4439.573409] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4439.576646] LustreError: 149078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4439.856963] Lustre: server umount lustre-MDT0001 complete [ 4451.587071] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 4454.662954] Key type lgssc unregistered [ 4455.104953] LNet: 154731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4455.112463] LNetError: 154731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4455.121759] LNet: Removed LNI 192.168.206.122@tcp [ 4456.027337] Key type .llcrypt unregistered [ 4456.029427] Key type ._llcrypt unregistered [ 4495.332953] Key type ._llcrypt registered [ 4495.335553] Key type .llcrypt registered [ 4495.447722] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 4496.499324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4496.554626] alg: No test for adler32 (adler32-zlib) [ 4497.723459] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 4498.072956] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 4499.823179] Key type lgssc registered [ 4501.250933] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4511.500299] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4511.533291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4513.186059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4518.653709] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 4523.488873] LustreError: 156252:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4523.551990] LustreError: 156252:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4528.612881] LustreError: 156253:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4531.022305] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4533.730730] LustreError: 156252:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4536.242978] LustreError: 156281:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4536.255820] LustreError: 156281:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4536.262263] LustreError: 156281:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 23, retries 0, failed: rc = -5 [ 4536.619526] Lustre: server umount lustre-MDT0000 complete [ 4556.125568] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [ 4565.870614] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 4610.792103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4611.120656] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4611.150193] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4611.242297] Lustre: lustre-MDT0000: new disk, initializing [ 4611.332718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4611.350345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4616.529063] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 4627.283807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4627.385378] Lustre: 159587:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4627.450394] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4627.461758] Lustre: Skipped 1 previous similar message [ 4627.605633] Lustre: lustre-MDT0001: new disk, initializing [ 4627.771697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4627.830393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4627.840817] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4632.655608] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 4641.643246] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4647.146508] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4653.239134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4653.555617] Lustre: lustre-OST0000: new disk, initializing [ 4653.566240] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4653.577231] Lustre: 161555:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4653.699773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4659.968450] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 4663.880847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4663.888771] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4663.965597] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4669.282717] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4673.170352] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4674.777986] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 09:35:49 (1789133749) [ 4945.376800] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4945.383445] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4945.393765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4945.890982] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4945.903116] Lustre: Skipped 1 previous similar message [ 4945.916837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4945.926325] Lustre: Skipped 1 previous similar message [ 4948.768935] Lustre: server umount lustre-MDT0000 complete [ 4951.015061] LustreError: 160813:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4951.044293] LustreError: 160813:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4951.955225] LustreError: 160382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789134027 with bad export cookie 7292724528896120006 [ 4951.961370] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4951.967880] LustreError: 160382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4956.128021] LustreError: 159596:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4956.132562] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4956.161842] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4961.248421] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4966.367140] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4966.751470] Lustre: server umount lustre-MDT0001 complete [ 4976.754805] Lustre: server umount lustre-OST0000 complete [ 4984.779538] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [ 4993.597791] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 5030.705323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5030.914090] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5030.934923] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5031.002915] Lustre: lustre-MDT0000: new disk, initializing [ 5031.065263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5031.076036] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5036.486988] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5046.690714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5046.838860] Lustre: 166225:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5046.881686] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5046.885570] Lustre: Skipped 1 previous similar message [ 5046.940780] Lustre: lustre-MDT0001: new disk, initializing [ 5047.020876] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5047.049688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5047.062690] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5051.120833] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5059.934370] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5065.702980] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5073.152169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5073.552679] Lustre: lustre-OST0000: new disk, initializing [ 5073.571504] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5073.582132] Lustre: 168193:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5073.702748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5080.136146] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5083.677565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5083.694751] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5083.798996] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5088.337903] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5092.440687] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5092.693908] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5096.600360] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5096.817586] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5099.495664] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5099.502529] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5099.515497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5099.519973] Lustre: Skipped 2 previous similar messages [ 5104.640287] LustreError: 168176:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5104.666778] LustreError: 168176:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 5104.924143] Lustre: server umount lustre-OST0000 complete [ 5109.731196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5109.751916] Lustre: Skipped 1 previous similar message [ 5109.760905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5109.777842] Lustre: Skipped 1 previous similar message [ 5115.304198] Lustre: server umount lustre-MDT0000 complete [ 5120.005971] LustreError: 167021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789134196 with bad export cookie 7292724528896278836 [ 5120.011852] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5120.030935] LustreError: 167021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5120.712334] Lustre: server umount lustre-MDT0001 complete [ 5132.352621] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:43:26 (1789134206) [ 5147.950679] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [ 5157.369718] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 5196.529620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5196.934284] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5196.956627] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5197.013964] Lustre: lustre-MDT0000: new disk, initializing [ 5197.062658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5197.083655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5202.218604] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5213.293461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5213.556326] Lustre: 172949:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5214.113184] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5218.717917] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5227.928968] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5234.700691] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5243.111752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5243.555110] Lustre: lustre-OST0000: new disk, initializing [ 5243.563916] Lustre: Skipped 1 previous similar message [ 5243.576213] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5243.581868] Lustre: Skipped 2 previous similar messages [ 5243.586117] Lustre: 174923:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5250.293216] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5253.730441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5253.739784] Lustre: Skipped 1 previous similar message [ 5253.749844] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5253.866457] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5261.957681] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5274.592661] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5274.599559] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5274.605525] LustreError: Skipped 2 previous similar messages [ 5274.606465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5274.606469] Lustre: Skipped 2 previous similar messages [ 5274.657443] Lustre: Skipped 2 previous similar messages [ 5276.617215] Lustre: server umount lustre-OST0000 complete [ 5280.736639] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5286.312560] Lustre: server umount lustre-MDT0000 complete [ 5289.563346] LustreError: 172940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789134365 with bad export cookie 7292724528896279718 [ 5289.572069] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5289.579986] LustreError: 172940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5289.825637] Lustre: server umount lustre-MDT0001 complete [ 5297.595283] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:46:12 (1789134372) [ 5313.528191] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [ 5323.667827] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 5361.795961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5362.022688] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5362.050408] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5362.126676] Lustre: lustre-MDT0000: new disk, initializing [ 5362.212474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5362.221171] Lustre: Skipped 2 previous similar messages [ 5362.233780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5366.734249] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5376.883265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5376.949805] Lustre: 179248:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5377.171859] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5382.524994] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5393.460663] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5399.149834] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5405.952263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5406.379883] Lustre: 181225:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5413.335557] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5414.472184] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5414.638923] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5424.428936] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5435.360032] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5435.368747] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5435.380575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5435.380582] Lustre: Skipped 4 previous similar messages [ 5435.416987] Lustre: Skipped 3 previous similar messages [ 5438.443626] Lustre: server umount lustre-OST0000 complete [ 5444.064671] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5448.829785] Lustre: server umount lustre-MDT0000 complete [ 5450.732340] LustreError: 179259:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5450.769126] LustreError: 179259:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 5453.854765] LustreError: 179240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789134529 with bad export cookie 7292724528896280803 [ 5453.869247] LustreError: 179240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5453.872522] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5454.258939] Lustre: server umount lustre-MDT0001 complete [ 5464.771471] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 5468.096950] Key type lgssc unregistered [ 5468.647718] LNet: 183085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5468.659684] LNetError: 183085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5469.735378] LNet: Removed LNI 192.168.206.122@tcp [ 5471.174463] Key type .llcrypt unregistered [ 5471.178708] Key type ._llcrypt unregistered [ 5499.939813] Key type ._llcrypt registered [ 5499.946789] Key type .llcrypt registered [ 5500.081908] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 5502.140267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5502.272182] alg: No test for adler32 (adler32-zlib) [ 5503.491468] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 5503.924064] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 5505.687818] Key type lgssc registered [ 5507.046027] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5516.340711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5516.356633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5517.916599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5522.102110] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5527.015105] LustreError: 184504:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5527.028689] LustreError: 184504:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5530.231952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5530.546198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5534.765548] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5543.021210] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5549.122152] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5556.517039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5557.040615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5562.179826] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5563.232155] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5571.490258] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5596.134714] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5596.136126] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5596.153527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5596.161992] Lustre: Skipped 1 previous similar message [ 5600.111105] Lustre: server umount lustre-OST0000 complete [ 5606.371102] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5606.392793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5606.399756] Lustre: Skipped 2 previous similar messages [ 5606.441889] Lustre: Skipped 1 previous similar message [ 5610.302528] Lustre: server umount lustre-MDT0000 complete [ 5611.490362] LustreError: 185522:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5614.998629] LustreError: 184484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789134691 with bad export cookie 5709017789828929333 [ 5615.003539] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5615.008348] LustreError: 184484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5615.515516] Lustre: server umount lustre-MDT0001 complete [ 5629.825418] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 5634.483333] Key type lgssc unregistered [ 5634.856508] LNet: 188065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5634.861423] LNetError: 188065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5634.885832] LNet: Removed LNI 192.168.206.122@tcp [ 5635.919141] Key type .llcrypt unregistered [ 5635.931745] Key type ._llcrypt unregistered [ 5666.308798] Key type ._llcrypt registered [ 5666.311375] Key type .llcrypt registered [ 5666.424712] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 5667.269786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5667.552409] alg: No test for adler32 (adler32-zlib) [ 5668.526800] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 5668.760264] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 5670.391450] Key type lgssc registered [ 5671.922262] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5681.254512] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5681.289726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5682.863311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5688.629680] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5692.385712] LustreError: 189483:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5692.430555] LustreError: 189483:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5697.521892] LustreError: 189484:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5699.579702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5700.203988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5704.711450] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5712.768363] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5720.294726] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5728.705054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5729.254652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5737.777445] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5747.680721] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5752.827853] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5760.997789] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5761.011335] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5761.036938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5763.042664] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5763.046287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5763.057139] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5763.100563] Lustre: Skipped 1 previous similar message [ 5765.231242] Lustre: lustre-OST0000: Not available for connect from 192.168.206.22@tcp (stopping) [ 5766.692585] Lustre: server umount lustre-OST0000 complete [ 5774.406695] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff898b99100800.ost_server_uuid 50 [ 5776.197399] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5792.224687] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5792.239296] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5792.264599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5797.119250] Lustre: server umount lustre-MDT0000 complete [ 5798.882387] LustreError: 189478:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5798.911376] LustreError: 189478:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5803.666211] LustreError: 189463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789134879 with bad export cookie 9887690506026114803 [ 5803.684404] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5804.185877] Lustre: server umount lustre-MDT0001 complete [ 5818.374954] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 09:54:51 (1789134891) [ 5824.822498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5825.262023] LustreError: 193421:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5825.336194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5829.971923] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5830.639268] LustreError: 193422:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5839.005281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5839.318619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5844.341670] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5856.475267] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5865.979927] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5874.950344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5875.284791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5882.869588] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5890.409850] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5896.842878] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5897.842955] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5898.074800] Lustre: lustre-OST0001: new disk, initializing [ 5898.078262] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5898.084529] Lustre: 196264:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5898.161872] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5905.554934] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5906.525350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5906.532402] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5906.677118] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5914.549411] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5917.666223] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5917.676772] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5917.708650] Lustre: Skipped 1 previous similar message [ 5917.717102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5917.734034] Lustre: Skipped 2 previous similar messages [ 5923.876969] Lustre: server umount lustre-OST0000 complete [ 5925.439812] LustreError: 195243:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5925.487490] LustreError: 195243:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5932.254148] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5932.682753] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5940.299278] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5940.555701] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5942.777669] LustreError: 196669:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5942.798678] LustreError: 196669:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 10 previous similar messages [ 5956.073218] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5956.090022] Lustre: Skipped 1 previous similar message [ 5956.096563] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5956.100964] Lustre: Skipped 5 previous similar messages [ 5960.516763] Lustre: server umount lustre-OST0001 complete [ 5966.307203] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5966.329692] Lustre: Skipped 2 previous similar messages [ 5970.731728] Lustre: server umount lustre-MDT0000 complete [ 5974.283205] LustreError: 193400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789135050 with bad export cookie 9887690506026115867 [ 5974.286272] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5974.295835] LustreError: 193400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5974.607649] Lustre: server umount lustre-MDT0001 complete [ 5998.442776] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 09:57:52 (1789135072) [ 6005.448346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6005.770079] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6005.797537] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6005.931891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6010.545825] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 6019.439638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6019.534965] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6019.601772] Lustre: 199708:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6024.222361] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 6032.813165] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6038.640481] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6045.612451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6045.865568] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6052.376711] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 6053.583603] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6058.840658] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6066.346598] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6066.446241] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6072.323578] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 6076.081216] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6079.634281] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6084.083369] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6084.101101] LustreError: Skipped 1 previous similar message [ 6084.105161] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6084.146181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6084.154107] Lustre: Skipped 2 previous similar messages [ 6088.469218] Lustre: server umount lustre-OST0000 complete [ 6090.303479] LustreError: 201560:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6090.325250] LustreError: 201560:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 10 previous similar messages [ 6107.114462] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6107.121862] LustreError: Skipped 2 previous similar messages [ 6111.816777] Lustre: server umount lustre-OST0001 complete [ 6117.348032] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6117.364708] Lustre: Skipped 3 previous similar messages [ 6117.373095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6117.388715] Lustre: Skipped 7 previous similar messages [ 6123.620631] Lustre: server umount lustre-MDT0000 complete [ 6127.623209] LustreError: 199701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789135203 with bad export cookie 9887690506026117281 [ 6127.628167] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6127.637795] LustreError: 199701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6128.192410] Lustre: server umount lustre-MDT0001 complete [ 6156.103321] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 10:00:29 (1789135229) [ 6179.672994] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [ 6190.347307] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 6228.728789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6229.043413] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6229.074773] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6229.185804] Lustre: lustre-MDT0000: new disk, initializing [ 6229.253020] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6229.263457] Lustre: Skipped 3 previous similar messages [ 6229.279777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6233.683583] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 6243.825485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6243.965746] Lustre: 207817:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6244.075097] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6244.079887] Lustre: Skipped 1 previous similar message [ 6244.172980] Lustre: lustre-MDT0001: new disk, initializing [ 6244.363522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6244.376354] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6249.929449] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 6265.075944] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6274.075370] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6282.497128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6282.786554] Lustre: lustre-OST0000: new disk, initializing [ 6282.791686] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6282.799893] Lustre: 209788:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6289.009326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6289.020635] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6289.143405] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6290.547371] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 6303.863904] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6309.532626] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6309.795072] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6316.542869] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6316.810728] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6320.101485] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6320.106969] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6320.118795] LustreError: Skipped 2 previous similar messages [ 6320.120952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6320.120959] Lustre: Skipped 2 previous similar messages [ 6320.148503] Lustre: Skipped 2 previous similar messages [ 6325.228130] LustreError: 210051:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6325.241273] LustreError: 210051:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 13 previous similar messages [ 6325.385596] Lustre: server umount lustre-OST0000 complete [ 6336.023091] Lustre: server umount lustre-MDT0000 complete [ 6340.812449] LustreError: 208611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789135416 with bad export cookie 9887690506026119066 [ 6340.825147] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6340.827549] LustreError: 208611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6341.646906] Lustre: server umount lustre-MDT0001 complete [ 6351.081978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6356.495019] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 6366.115358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6366.668789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6366.684552] Lustre: Skipped 3 previous similar messages [ 6371.763095] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 6380.902913] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6387.738917] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6396.999414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6406.736780] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 6418.046454] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6425.723463] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6425.952892] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6430.849351] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6431.038899] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6436.832555] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6439.253542] Lustre: server umount lustre-OST0000 complete [ 6445.885514] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6450.745247] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6482.723754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6490.354310] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 6500.013957] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6501.572941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6501.590138] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6501.592365] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6502.007300] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6506.976246] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6506.980226] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6506.984794] LustreError: Skipped 1 previous similar message [ 6506.991332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6506.991339] Lustre: Skipped 7 previous similar messages [ 6507.046682] Lustre: Skipped 5 previous similar messages [ 6512.976950] Lustre: server umount lustre-OST0000 complete [ 6524.681294] Lustre: server umount lustre-MDT0000 complete [ 6525.411036] LustreError: 212991:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6525.475066] LustreError: 212991:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 6529.997524] LustreError: 211988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789135606 with bad export cookie 9887690506026119955 [ 6530.024404] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6530.678047] Lustre: server umount lustre-MDT0001 complete [ 6543.645656] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 10:06:57 (1789135617) [ 6550.975661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6556.060712] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 6565.942501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6571.178665] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 6581.443453] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6591.399092] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6601.949590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6610.264265] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 6619.444996] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6626.338985] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6626.625164] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6633.287096] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6633.658462] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6643.005227] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6643.258143] Lustre: lustre-OST0001: new disk, initializing [ 6643.264169] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6643.269863] Lustre: 221345:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6643.371476] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6643.381417] Lustre: Skipped 5 previous similar messages [ 6650.616243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6650.627261] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6650.753268] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6651.146069] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 6662.544698] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6666.666196] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6666.926328] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6670.887659] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6671.135297] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6676.453377] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6676.459092] LustreError: Skipped 1 previous similar message [ 6679.569026] Lustre: server umount lustre-OST0001 complete [ 6688.672785] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6689.110628] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6695.315155] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6695.537753] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6726.180780] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6731.746859] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6731.764746] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6731.766907] Lustre: Skipped 1 previous similar message [ 6731.768044] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6732.908772] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 6740.199457] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6741.871435] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6757.856626] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6758.060379] Lustre: server umount lustre-OST0001 complete [ 6775.776375] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6775.779841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6775.787620] Lustre: Skipped 7 previous similar messages [ 6775.791018] Lustre: Skipped 13 previous similar messages [ 6781.178772] Lustre: server umount lustre-OST0000 complete [ 6799.327371] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6799.483458] Lustre: server umount lustre-MDT0000 complete [ 6802.405627] LustreError: 217974:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6802.432912] LustreError: 217974:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 36 previous similar messages [ 6803.174109] LustreError: 217953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789135879 with bad export cookie 9887690506026120921 [ 6803.192256] LustreError: 217953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6803.195447] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6809.563841] Lustre: server umount lustre-MDT0001 complete [ 6833.353421] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 10:11:47 (1789135907) [ 6840.216112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6840.659896] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6840.695549] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6847.137464] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 6856.524645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6856.688625] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6856.748678] Lustre: 226590:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6861.947555] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 6869.632857] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6875.905182] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6882.799818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6882.926627] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6889.311924] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 6897.098583] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6907.648775] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6907.725758] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6914.100461] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 6922.671847] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6924.593771] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6929.254987] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6929.429719] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6934.122272] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6934.313609] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6940.549591] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff898b86d74000.ost_server_uuid 50 [ 6942.089132] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff898b86d74000.ost_server_uuid in IDLE state after 0 sec [ 6943.004841] Lustre: Permanently deactivating lustre-OST0001 [ 6943.022520] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6946.644272] Lustre: Permanently reactivating lustre-OST0001 [ 6946.659662] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6946.667923] Lustre: Skipped 2 previous similar messages [ 6952.422954] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6952.431966] LustreError: Skipped 5 previous similar messages [ 6956.143270] Lustre: server umount lustre-OST0001 complete [ 6965.813243] Lustre: server umount lustre-OST0000 complete [ 6975.583421] Lustre: server umount lustre-MDT0000 complete [ 6979.691076] LustreError: 226584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136055 with bad export cookie 9887690506026122209 [ 6979.704698] LustreError: 226584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6979.705167] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6980.023965] Lustre: server umount lustre-MDT0001 complete [ 7006.068797] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 10:14:40 (1789136080) [ 7014.090594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7014.375315] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7014.409705] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7014.419799] Lustre: Skipped 2 previous similar messages [ 7019.652715] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 7028.846453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7029.036029] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7029.072567] Lustre: 233287:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7033.771415] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 7040.853684] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7046.887239] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7054.259310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7054.403070] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7054.418994] Lustre: Permanently deactivating lustre-OST0000 [ 7054.429552] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7060.933905] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 7064.109064] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7064.224174] LustreError: 235639:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -108 [ 7068.598401] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7074.842434] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7074.912610] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7081.702604] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7082.005264] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 7090.195700] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7092.245420] Lustre: Permanently reactivating lustre-OST0000 [ 7092.257301] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7092.262780] Lustre: Skipped 2 previous similar messages [ 7102.032607] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7102.048526] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7102.068699] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7102.074202] Lustre: Skipped 1 previous similar message [ 7102.131559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7102.151804] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7110.012446] Lustre: server umount lustre-OST0001 complete [ 7128.033538] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7128.187412] Lustre: server umount lustre-OST0000 complete [ 7138.359250] Lustre: server umount lustre-MDT0000 complete [ 7141.651438] LustreError: 234027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136217 with bad export cookie 9887690506026123889 [ 7141.667042] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7141.668939] LustreError: 234027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7141.933734] Lustre: server umount lustre-MDT0001 complete [ 7153.475081] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 7156.757547] Key type lgssc unregistered [ 7157.118662] LNet: 238174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7157.135088] LNetError: 238174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7157.157204] LNet: Removed LNI 192.168.206.122@tcp [ 7158.238570] Key type .llcrypt unregistered [ 7158.249432] Key type ._llcrypt unregistered [ 7183.073945] Key type ._llcrypt registered [ 7183.079286] Key type .llcrypt registered [ 7183.293184] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 7184.685419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7184.876733] alg: No test for adler32 (adler32-zlib) [ 7186.108618] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 7186.426529] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 7188.223163] Key type lgssc registered [ 7189.269042] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7199.095456] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7199.118793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7200.780380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7204.652487] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 7210.984875] LustreError: 239642:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7211.014306] LustreError: 239642:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7212.781837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7212.900855] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7212.935941] Lustre: 239633:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7212.946492] Lustre: Permanently deactivating lustre-MDT0001 [ 7212.956955] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7213.216508] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7217.169402] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 7223.339592] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7223.438906] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7228.693899] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7235.172561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7235.393750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7238.437839] LustreError: 241481:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7241.830427] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 7243.752377] LustreError: 241661:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7245.798539] LustreError: 241484:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7245.828822] LustreError: 241484:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7245.846632] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7249.835892] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7250.916520] LustreError: 241661:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7250.935586] LustreError: 241661:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7256.279707] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7256.542623] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7261.689425] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7261.702347] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7262.597318] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 7270.621993] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7274.795153] Lustre: Permanently reactivating lustre-MDT0001 [ 7274.829431] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7274.841592] Lustre: Skipped 2 previous similar messages [ 7280.746521] Lustre: lustre-MDT0001: Client f67b4e2b-7e97-43b1-9b30-a62e8ae572b7 (at 192.168.206.22@tcp) reconnecting [ 7282.808278] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7282.830099] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7282.844208] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7282.867241] LustreError: 243214:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7282.867714] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7282.880822] LustreError: 243214:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7282.880933] LustreError: 243214:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.122@tcp: cfg command failed: rc = -108 [ 7282.881050] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7282.881050] [ 7282.882055] LustreError: 239635:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7287.345966] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7289.008214] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7294.497435] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7294.729632] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7298.991297] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7299.201498] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7301.693811] Lustre: Permanently deactivating lustre-MDT0001 [ 7301.708914] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7301.719240] Lustre: Skipped 2 previous similar messages [ 7310.944936] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7320.546412] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7320.563537] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7320.580668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7324.130309] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7324.137146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7324.165674] Lustre: Skipped 2 previous similar messages [ 7324.190767] Lustre: Skipped 3 previous similar messages [ 7326.684560] Lustre: server umount lustre-MDT0000 complete [ 7329.249548] LustreError: 240386:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7329.281279] LustreError: 240386:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 7330.499969] LustreError: 239625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136406 with bad export cookie 11767590096149028215 [ 7330.506042] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7330.522316] LustreError: 239625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7330.866795] Lustre: server umount lustre-MDT0001 complete [ 7336.319550] Lustre: server umount lustre-OST0000 complete [ 7349.158721] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 10:20:23 (1789136423) [ 7364.395679] Lustre: server umount lustre-OST0001 complete [ 7372.201024] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [ 7382.324637] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 7417.125701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7417.490412] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7417.495225] Lustre: Skipped 2 previous similar messages [ 7417.517774] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7417.628324] Lustre: lustre-MDT0000: new disk, initializing [ 7417.747970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7417.779455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7423.798485] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 7435.708211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7435.802373] Lustre: 247947:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7435.895299] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7435.898799] Lustre: Skipped 1 previous similar message [ 7435.969314] Lustre: lustre-MDT0001: new disk, initializing [ 7436.087736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7436.156821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7436.176130] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7442.489171] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 7453.150504] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7459.170751] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7466.609389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7466.852524] Lustre: lustre-OST0000: new disk, initializing [ 7466.855885] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7466.862715] Lustre: 249917:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7466.934520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7473.608517] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 7474.796357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7474.814354] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7474.974866] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7482.045947] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7483.934862] LustreError: 249325:0:(mdt_handler.c:3282:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7491.817748] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7492.018510] Lustre: lustre-OST0001: new disk, initializing [ 7492.022289] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7492.039756] Lustre: 251030:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7493.952221] LustreError: 249325:0:(mdt_handler.c:3282:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7493.972891] LustreError: 247955:0:(mdt_handler.c:3282:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7497.526587] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 7501.896352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7501.905165] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7501.984060] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7504.039084] LustreError: 247955:0:(mdt_handler.c:3282:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7506.033970] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7507.403254] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7509.087517] Lustre: lustre-OST0001: Not available for connect from 192.168.206.22@tcp (stopping) [ 7512.546451] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7512.550716] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7512.554490] LustreError: Skipped 1 previous similar message [ 7512.555633] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7512.560848] Lustre: Skipped 1 previous similar message [ 7517.675540] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7517.682393] Lustre: Skipped 3 previous similar messages [ 7523.295326] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7523.389589] Lustre: server umount lustre-OST0001 complete [ 7524.413217] LustreError: 249901:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7541.728821] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7541.734754] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7541.739477] LustreError: Skipped 1 previous similar message [ 7541.740844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7541.761211] Lustre: Skipped 1 previous similar message [ 7541.769961] Lustre: Skipped 4 previous similar messages [ 7545.568552] Lustre: server umount lustre-OST0000 complete [ 7554.586710] Lustre: server umount lustre-MDT0000 complete [ 7557.376808] LustreError: 247941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136633 with bad export cookie 11767590096149030812 [ 7557.381930] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7557.386862] LustreError: 247941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7563.923587] Lustre: server umount lustre-MDT0001 complete [ 7571.778989] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 7573.980436] Key type lgssc unregistered [ 7574.207678] LNet: 253067:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7574.217375] LNetError: 253067:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7574.238678] LNet: Removed LNI 192.168.206.122@tcp [ 7574.838230] Key type .llcrypt unregistered [ 7574.840800] Key type ._llcrypt unregistered [ 7605.470921] Key type ._llcrypt registered [ 7605.472723] Key type .llcrypt registered [ 7605.552057] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 7606.658460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7606.674024] alg: No test for adler32 (adler32-zlib) [ 7607.683092] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 7607.867388] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 7609.503121] Key type lgssc registered [ 7610.372561] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7617.682619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7617.698362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7618.931943] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7618.964473] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7619.131695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7623.425095] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 7631.156735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7631.253036] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7631.291612] Lustre: 255252:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7631.538660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7635.269524] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 7641.741374] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7646.425401] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7651.720570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7651.867396] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7652.032490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7657.284512] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 7661.235134] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7664.331976] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7717.355964] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7717.361070] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7717.372688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7717.859748] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7717.864389] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7717.886779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7720.387772] Lustre: server umount lustre-OST0000 complete [ 7724.574676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7736.884519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7737.430752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7742.446157] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7742.459577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7742.459851] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7742.490685] Lustre: Skipped 1 previous similar message [ 7743.654675] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 7750.082419] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7788.512788] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7788.517358] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7788.523757] LustreError: Skipped 1 previous similar message [ 7788.524658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7788.557212] Lustre: Skipped 1 previous similar message [ 7793.633972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7793.641673] Lustre: Skipped 2 previous similar messages [ 7794.167704] Lustre: server umount lustre-OST0000 complete [ 7798.753104] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7798.763897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7798.764364] Lustre: Skipped 1 previous similar message [ 7798.776837] Lustre: Skipped 1 previous similar message [ 7803.301982] Lustre: server umount lustre-MDT0000 complete [ 7805.822565] LustreError: 257116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136881 with bad export cookie 2501330272620411934 [ 7805.828247] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7805.838416] LustreError: 257116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7806.054229] Lustre: server umount lustre-MDT0001 complete [ 7815.455171] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 7817.953368] Key type lgssc unregistered [ 7818.244916] LNet: 260683:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7818.256772] LNetError: 260683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7818.290913] LNet: Removed LNI 192.168.206.122@tcp [ 7819.007957] Key type .llcrypt unregistered [ 7819.009556] Key type ._llcrypt unregistered [ 7842.317726] Key type ._llcrypt registered [ 7842.322663] Key type .llcrypt registered [ 7842.482607] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 7843.551204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7843.757614] alg: No test for adler32 (adler32-zlib) [ 7844.860817] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 7845.129583] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 7846.799878] Key type lgssc registered [ 7848.199533] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7855.467946] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7855.478917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7856.879970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7860.818660] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 7866.348439] LustreError: 262098:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7866.362330] LustreError: 262098:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7868.041636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7868.374250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7872.676154] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 7879.114165] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7884.692809] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7890.943246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7891.172569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7895.464125] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 7901.442043] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7901.700719] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7922.148837] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7922.153585] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7922.154273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7922.164449] Lustre: Skipped 1 previous similar message [ 7926.774645] Lustre: server umount lustre-OST0000 complete [ 7929.825068] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7929.833526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7929.847258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7929.861973] Lustre: Skipped 1 previous similar message [ 7932.401880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7932.404398] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7935.863443] Lustre: server umount lustre-MDT0000 complete [ 7937.506881] LustreError: 262826:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7938.793039] LustreError: 262079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137014 with bad export cookie 4947518970713873312 [ 7938.797246] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7938.805368] LustreError: 262079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7939.057210] Lustre: server umount lustre-MDT0001 complete [ 7947.024960] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 7949.497564] Key type lgssc unregistered [ 7949.801815] LNet: 266404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7949.811708] LNetError: 266404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7949.826265] LNet: Removed LNI 192.168.206.122@tcp [ 7950.599087] Key type .llcrypt unregistered [ 7950.601800] Key type ._llcrypt unregistered [ 7968.019758] Key type ._llcrypt registered [ 7968.022205] Key type .llcrypt registered [ 7968.116275] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 7969.373930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7969.424548] alg: No test for adler32 (adler32-zlib) [ 7970.397740] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 7970.736167] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 7972.415712] Key type lgssc registered [ 7973.563133] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7981.139765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7981.176497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7982.662165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7987.263949] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 7992.801033] LustreError: 267627:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7992.813606] LustreError: 267627:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7995.531710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7995.923412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8000.628477] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 8008.632042] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8014.071815] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8019.933066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8020.274659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8025.436757] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 8027.512053] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8032.444182] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8042.983996] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8042.984540] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8042.986180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8042.995961] Lustre: Skipped 1 previous similar message [ 8048.096433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8048.100597] Lustre: Skipped 1 previous similar message [ 8048.339490] Lustre: server umount lustre-OST0000 complete [ 8052.192633] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8052.200522] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8052.221841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8053.218056] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8057.792549] Lustre: server umount lustre-MDT0000 complete [ 8058.336221] LustreError: 268355:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8058.351652] LustreError: 268355:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8061.103826] LustreError: 267607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137137 with bad export cookie 5222355192211678180 [ 8061.105040] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8061.113907] LustreError: 267607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8061.649428] Lustre: server umount lustre-MDT0001 complete [ 8071.273883] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 8073.711435] Key type lgssc unregistered [ 8073.981564] LNet: 271312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8073.996791] LNetError: 271312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8074.006908] LNet: Removed LNI 192.168.206.122@tcp [ 8074.697780] Key type .llcrypt unregistered [ 8074.699800] Key type ._llcrypt unregistered [ 8098.938124] Key type ._llcrypt registered [ 8098.942267] Key type .llcrypt registered [ 8099.045550] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 8099.910982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8100.043127] alg: No test for adler32 (adler32-zlib) [ 8101.088525] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 8101.385363] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 8103.063166] Key type lgssc registered [ 8104.078975] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8111.589862] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8111.623206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8113.041033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8117.150934] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 8123.362908] LustreError: 272731:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8123.372164] LustreError: 272731:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8124.432059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8124.777085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8128.331565] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 8134.781266] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8139.332329] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8145.231414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8145.556649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8151.421040] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 8157.755682] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8158.906383] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8179.684208] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8179.690449] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8179.697324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8179.720381] Lustre: Skipped 1 previous similar message [ 8184.414787] Lustre: server umount lustre-OST0000 complete [ 8189.921479] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8189.926459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8189.937062] Lustre: Skipped 2 previous similar messages [ 8189.946755] Lustre: Skipped 1 previous similar message [ 8195.041351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8195.049140] Lustre: Skipped 1 previous similar message [ 8199.139029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8199.149417] Lustre: Skipped 1 previous similar message [ 8202.207125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8202.346266] Lustre: server umount lustre-MDT0000 complete [ 8204.257345] LustreError: 272731:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8204.268492] LustreError: 272731:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8205.462747] LustreError: 272711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137281 with bad export cookie 3883348576701621539 [ 8205.474011] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8205.743615] Lustre: server umount lustre-MDT0001 complete [ 8214.719472] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 8217.384306] Key type lgssc unregistered [ 8217.737830] LNet: 277039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8217.749981] LNetError: 277039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8217.778340] LNet: Removed LNI 192.168.206.122@tcp [ 8218.934760] Key type .llcrypt unregistered [ 8218.947621] Key type ._llcrypt unregistered [ 8237.254942] Key type ._llcrypt registered [ 8237.262052] Key type .llcrypt registered [ 8237.374444] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 8238.442830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8238.463716] alg: No test for adler32 (adler32-zlib) [ 8239.523050] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 8239.787641] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 8241.431237] Key type lgssc registered [ 8242.653816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8250.360623] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8250.379890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8251.831610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8255.399242] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 8261.096680] LustreError: 278273:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8261.111925] LustreError: 278273:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8262.856725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8263.230986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8267.789468] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 8275.484899] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8282.214319] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8289.977241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8290.360947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8296.498509] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8296.767747] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 8304.201679] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8318.951869] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8318.952202] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8318.982440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8318.992184] Lustre: Skipped 1 previous similar message [ 8319.012333] Lustre: Skipped 1 previous similar message [ 8323.645831] Lustre: server umount lustre-OST0000 complete [ 8329.184767] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8329.190255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8329.204672] Lustre: Skipped 1 previous similar message [ 8329.215401] Lustre: Skipped 1 previous similar message [ 8333.140834] Lustre: server umount lustre-MDT0000 complete [ 8334.308842] LustreError: 278260:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8334.335524] LustreError: 278260:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8336.412841] LustreError: 279032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137412 with bad export cookie 11077432527784347161 [ 8336.421390] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8336.422571] LustreError: 279032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8336.706308] Lustre: server umount lustre-MDT0001 complete [ 8345.160672] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 8347.481952] Key type lgssc unregistered [ 8347.681725] LNet: 281977:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8347.689677] LNetError: 281977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8348.709855] LNet: Removed LNI 192.168.206.122@tcp [ 8349.491127] Key type .llcrypt unregistered [ 8349.495803] Key type ._llcrypt unregistered [ 8359.870583] Key type ._llcrypt registered [ 8359.872768] Key type .llcrypt registered [ 8359.969908] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8374.387332] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [ 8387.624608] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 8388.859452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8388.884465] alg: No test for adler32 (adler32-zlib) [ 8389.827687] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 8390.028560] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 8391.703238] Key type lgssc registered [ 8392.775033] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8428.155113] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8428.175822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8429.403873] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8429.432987] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8429.487302] Lustre: lustre-MDT0000: new disk, initializing [ 8429.544828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8429.558832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8433.276989] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 8442.926509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8443.022796] Lustre: 285512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8443.070276] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8443.074286] Lustre: Skipped 1 previous similar message [ 8443.130409] Lustre: lustre-MDT0001: new disk, initializing [ 8443.203977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8443.239644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8443.248672] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8447.318776] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 8457.795119] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8464.028924] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8472.002551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8472.200675] Lustre: lustre-OST0000: new disk, initializing [ 8472.203638] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8472.209602] Lustre: 287481:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8472.291207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8478.784549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8478.802838] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8478.954690] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8479.182959] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 8488.863885] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8493.510408] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8493.746609] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8498.561752] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8498.756408] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8504.802942] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8504.808104] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8504.812853] LustreError: Skipped 1 previous similar message [ 8504.823283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8504.826991] Lustre: Skipped 1 previous similar message [ 8504.842064] Lustre: Skipped 1 previous similar message [ 8507.137291] Lustre: server umount lustre-OST0000 complete [ 8515.051579] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8515.053652] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8515.057362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8515.070440] Lustre: Skipped 1 previous similar message [ 8516.721807] Lustre: server umount lustre-MDT0000 complete [ 8520.074482] LustreError: 285504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137596 with bad export cookie 5268480629243477906 [ 8520.077690] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8520.090276] LustreError: 285504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8520.424901] Lustre: server umount lustre-MDT0001 complete [ 8528.035935] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 10:40:02 (1789137602) [ 8534.681237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8535.186398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8539.708643] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 8540.648660] LustreError: 289890:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8540.667309] LustreError: 289890:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8545.781891] LustreError: 289889:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8547.202983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8547.586404] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8550.947899] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 8556.761392] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8561.035825] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8566.035652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8566.409415] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8571.786676] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 8578.575304] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8593.376546] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8593.385149] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8593.407475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8593.413806] Lustre: Skipped 1 previous similar message [ 8596.448514] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8596.451386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8596.467542] Lustre: Skipped 1 previous similar message [ 8597.753670] Lustre: server umount lustre-OST0000 complete [ 8601.574896] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8601.590932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8601.593152] Lustre: Skipped 1 previous similar message [ 8607.748146] Lustre: server umount lustre-MDT0000 complete [ 8610.933914] LustreError: 289870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137686 with bad export cookie 5268480629243478781 [ 8610.938718] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8610.943261] LustreError: 289870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8611.235823] Lustre: server umount lustre-MDT0001 complete [ 8620.700632] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 8623.498988] Key type lgssc unregistered [ 8623.846152] LNet: 293447:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8623.859151] LNetError: 293447:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8623.881184] LNet: Removed LNI 192.168.206.122@tcp [ 8624.817178] Key type .llcrypt unregistered [ 8624.826104] Key type ._llcrypt unregistered [ 8654.015670] Key type ._llcrypt registered [ 8654.021253] Key type .llcrypt registered [ 8654.187944] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [ 8668.779431] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 8670.025813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8670.198119] alg: No test for adler32 (adler32-zlib) [ 8671.246521] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 8671.452036] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 8673.151708] Key type lgssc registered [ 8674.317230] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8718.507396] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8718.542818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8719.791455] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8719.838045] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8719.955930] Lustre: lustre-MDT0000: new disk, initializing [ 8720.065870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8720.103987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8724.448840] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 8735.367895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8735.503593] Lustre: 297274:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8735.556689] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8735.564928] Lustre: Skipped 1 previous similar message [ 8735.669645] Lustre: lustre-MDT0001: new disk, initializing [ 8735.761388] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8735.801794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8735.819490] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8740.394485] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 8750.349260] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8756.470198] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8764.402231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8764.684985] Lustre: lustre-OST2710: new disk, initializing [ 8764.690556] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8764.706864] Lustre: 299242:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8764.851978] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8770.738443] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 8772.180130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8772.203680] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8772.368765] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8780.443881] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8786.702921] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8786.874693] Lustre: lustre-OST03e8: new disk, initializing [ 8786.877696] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8786.882912] Lustre: 300306:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8786.984653] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8792.808290] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 8793.127156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8793.137696] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8793.249776] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8798.833731] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8805.706066] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8805.900334] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8809.883276] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8810.109749] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8814.249426] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8814.484577] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8820.184860] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8820.426580] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8827.874755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8827.877365] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8827.895604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8829.410410] Lustre: lustre-MDT0000-lwp-OST03e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8829.419040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8829.439130] Lustre: Skipped 1 previous similar message [ 8829.471939] Lustre: Skipped 3 previous similar messages [ 8833.434901] Lustre: server umount lustre-MDT0000 complete [ 8836.696526] LustreError: 302133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137912 with bad export cookie 15357908456442560638 [ 8836.703835] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8836.707442] LustreError: 302133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8837.057665] Lustre: server umount lustre-MDT0001 complete [ 8852.409311] Lustre: server umount lustre-OST2710 complete [ 8855.526851] Lustre: 295073:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789137915/real 1789137915] req@00000000272b7813 x1876046902264960/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789137931 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8855.550811] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8855.569917] Lustre: Skipped 1 previous similar message [ 8855.831879] Lustre: server umount lustre-OST03e8 complete [ 8872.987269] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [ 8881.600951] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 8910.410535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8910.577722] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8910.612855] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8910.688547] Lustre: lustre-MDT0000: new disk, initializing [ 8910.751046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8910.762831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8914.645823] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 8924.214943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8924.325645] Lustre: 305556:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8924.386870] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8924.390638] Lustre: Skipped 1 previous similar message [ 8924.447764] Lustre: lustre-MDT0001: new disk, initializing [ 8924.565875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8924.606626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8924.616190] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8928.550475] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 8936.877414] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8941.200677] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8945.710513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8945.906546] Lustre: lustre-OST0000: new disk, initializing [ 8945.911277] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8945.917925] Lustre: 307526:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8945.996863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8950.702741] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 8952.400766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8952.418098] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8952.500711] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8958.675274] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8962.161681] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8962.330235] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8966.051900] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8966.244263] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8968.162046] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8968.163607] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8968.167496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8968.173443] LustreError: Skipped 1 previous similar message [ 8968.189114] Lustre: Skipped 1 previous similar message [ 8973.287440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8973.290222] Lustre: Skipped 2 previous similar messages [ 8974.292600] Lustre: server umount lustre-OST0000 complete [ 8978.402774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8978.404317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8978.415710] Lustre: Skipped 1 previous similar message [ 8978.425901] Lustre: Skipped 1 previous similar message [ 8984.316245] Lustre: server umount lustre-MDT0000 complete [ 8987.467993] LustreError: 305547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789138063 with bad export cookie 15357908456442562332 [ 8987.475490] LustreError: 305547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8987.475523] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8987.762394] Lustre: server umount lustre-MDT0001 complete [ 8993.960391] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 10:47:49 (1789138069) [ 8995.346784] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8996.659716] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 10:47:51 (1789138071) [ 8999.779371] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 9021.155607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9021.376371] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9021.389106] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9021.495140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9024.864734] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9031.687642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9031.834184] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9031.875831] Lustre: 311381:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9035.709733] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9055.718304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9055.883908] Lustre: Denying initial registration attempt from nid 192.168.206.122@tcp, specified as failover [ 9055.893395] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9055.901472] LustreError: 312833:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9055.907896] LustreError: 312833:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9055.912587] LustreError: 312833:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9055.963896] Lustre: server umount lustre-OST0000 complete [ 9055.967195] LustreError: 312833:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9062.880985] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9062.888223] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9062.897257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9062.901672] Lustre: Skipped 2 previous similar messages [ 9067.143783] Lustre: server umount lustre-MDT0000 complete [ 9069.538078] LustreError: 311391:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9069.553720] LustreError: 311373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789138145 with bad export cookie 15357908456442563214 [ 9069.554286] LustreError: 311391:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 9069.556677] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9069.562215] LustreError: 311373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9069.788842] Lustre: server umount lustre-MDT0001 complete [ 9076.918071] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 9079.075348] Key type lgssc unregistered [ 9079.292903] LNet: 313957:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9079.306872] LNetError: 313957:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9079.332564] LNet: Removed LNI 192.168.206.122@tcp [ 9079.898183] Key type .llcrypt unregistered [ 9079.903732] Key type ._llcrypt unregistered [ 9088.375301] Key type ._llcrypt registered [ 9088.378614] Key type .llcrypt registered [ 9088.486937] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 9089.126268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9089.170098] alg: No test for adler32 (adler32-zlib) [ 9090.141384] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 9090.297625] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 9091.919169] Key type lgssc registered [ 9092.786730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9131.548552] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 9138.065779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9138.090462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9139.327898] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9139.357291] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9139.534031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9143.893737] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9151.218626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9151.367738] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9151.426842] Lustre: 316702:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9151.716871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9155.451230] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9161.925066] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9166.786667] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9185.586590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9185.863532] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9186.193702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9194.191734] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9202.864974] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9209.314085] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9209.330117] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9209.341463] LustreError: Skipped 1 previous similar message [ 9209.349330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9209.363040] Lustre: Skipped 1 previous similar message [ 9210.923208] Lustre: server umount lustre-OST0000 complete [ 9214.434291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9214.453263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9214.459168] Lustre: Skipped 2 previous similar messages [ 9219.554470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9219.568020] Lustre: Skipped 1 previous similar message [ 9220.668614] Lustre: server umount lustre-MDT0000 complete [ 9223.790512] LustreError: 316692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789138299 with bad export cookie 6212247715572718083 [ 9223.794324] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9223.802312] LustreError: 316692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9224.109425] Lustre: server umount lustre-MDT0001 complete [ 9234.058946] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 9236.434446] Key type lgssc unregistered [ 9236.725713] LNet: 320278:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9236.737823] LNetError: 320278:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9236.753900] LNet: Removed LNI 192.168.206.122@tcp [ 9237.553336] Key type .llcrypt unregistered [ 9237.557217] Key type ._llcrypt unregistered [ 9261.932730] Key type ._llcrypt registered [ 9261.935483] Key type .llcrypt registered [ 9262.025919] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 9262.826772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9262.940220] alg: No test for adler32 (adler32-zlib) [ 9263.996509] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 9264.241667] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 9265.901957] Key type lgssc registered [ 9266.926811] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9274.477161] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9274.493442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9275.889496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9279.765264] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9285.098250] LustreError: 321695:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9285.104689] LustreError: 321695:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9287.729502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9288.367016] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9293.526517] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9301.733235] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9308.259339] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9317.938870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9318.226680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9325.368648] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9332.681752] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9343.973766] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9343.975555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9343.983383] Lustre: Skipped 1 previous similar message [ 9345.638515] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.22@tcp (stopping) [ 9345.641771] Lustre: Skipped 2 previous similar messages [ 9348.059256] Lustre: server umount lustre-MDT0000 complete [ 9349.097315] LustreError: 321690:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9349.125694] LustreError: 321690:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9350.725253] LustreError: 321689:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9352.726596] LustreError: 323539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789138428 with bad export cookie 10383266399636469721 [ 9352.749190] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9352.754539] LustreError: 323539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9354.208569] LustreError: 321690:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9354.209927] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9354.273045] Lustre: Skipped 1 previous similar message [ 9354.282148] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9357.218838] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9357.223760] Lustre: Skipped 1 previous similar message [ 9359.186833] Lustre: server umount lustre-MDT0001 complete [ 9365.115434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9372.384151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9377.769342] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x9018c5c70e94d5d1 [ 9377.786974] Lustre: MGC192.168.206.122@tcp: Connection restored to 0@lo (at 0@lo) [ 9378.031801] LustreError: 323527:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9378.046937] LustreError: 323527:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9378.103889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9378.171811] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9382.323680] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9387.299095] LustreError: 325290:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9387.309910] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9387.315455] LustreError: 325290:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 9389.992369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9390.506641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9394.728556] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9395.696589] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9402.186179] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9404.030078] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9408.499138] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9410.361063] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9414.112708] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9414.118264] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9414.122894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9416.165535] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9416.173276] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9420.271589] Lustre: server umount lustre-OST0000 complete [ 9426.402481] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9426.409852] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9426.417715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9426.443870] Lustre: Skipped 1 previous similar message [ 9426.454963] Lustre: Skipped 3 previous similar messages [ 9430.385892] Lustre: server umount lustre-MDT0000 complete [ 9431.521426] LustreError: 325294:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9431.555913] LustreError: 325294:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9434.410307] LustreError: 323539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789138510 with bad export cookie 10383266399636477393 [ 9434.417441] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9434.430541] LustreError: 323539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9435.127707] Lustre: server umount lustre-MDT0001 complete [ 9447.243332] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 9450.050778] Key type lgssc unregistered [ 9450.474664] LNet: 327965:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9450.480305] LNetError: 327965:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9450.501314] LNet: Removed LNI 192.168.206.122@tcp [ 9451.226830] Key type .llcrypt unregistered [ 9451.230931] Key type ._llcrypt unregistered [ 9473.272172] Key type ._llcrypt registered [ 9473.280254] Key type .llcrypt registered [ 9473.377872] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 9474.524701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9474.576708] alg: No test for adler32 (adler32-zlib) [ 9475.606817] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 9475.801762] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 9477.435168] Key type lgssc registered [ 9478.373508] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9485.703420] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9485.722485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9487.145031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9491.557931] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9497.573827] LustreError: 329382:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9497.612961] LustreError: 329382:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9500.829916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9501.230483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9506.488649] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9514.207735] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9520.700812] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9526.127682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9526.132297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9526.147080] Lustre: Skipped 1 previous similar message [ 9526.168949] Lustre: Skipped 1 previous similar message [ 9530.351745] Lustre: server umount lustre-MDT0000 complete [ 9531.369104] LustreError: 329383:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9531.406833] LustreError: 329383:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9534.755603] LustreError: 329364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789138610 with bad export cookie 13052812020825270725 [ 9534.764139] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9534.770158] LustreError: 329364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9535.033252] Lustre: server umount lustre-MDT0001 complete [ 9551.691365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9552.202557] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9552.249740] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9552.644993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9556.596522] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9565.343777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9565.531598] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9565.570605] Lustre: 331969:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9565.901369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9570.545571] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9578.143500] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9583.658496] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9593.322765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9593.510530] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9593.522198] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9593.528370] LustreError: 333857:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9593.536256] LustreError: 333857:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9593.549545] LustreError: 333857:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9593.647964] Lustre: server umount lustre-OST0000 complete [ 9593.650771] LustreError: 333857:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9603.880607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9604.102591] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9604.316468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9610.476808] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9614.500842] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9618.552240] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9626.040473] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9626.111459] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9626.195659] Lustre: lustre-OST0001: new disk, initializing [ 9626.199532] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9626.212607] Lustre: 335399:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9626.298420] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9632.729742] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9632.929142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9632.944329] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9633.080176] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9640.455948] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9643.497176] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9643.506706] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9643.513958] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9643.517869] LustreError: Skipped 1 previous similar message [ 9648.613773] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9648.631924] Lustre: Skipped 2 previous similar messages [ 9649.577537] Lustre: server umount lustre-OST0001 complete [ 9653.731210] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9653.731433] LustreError: 334325:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9653.749550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9653.754969] Lustre: Skipped 2 previous similar messages [ 9653.802482] LustreError: 334325:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9658.874257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9658.876647] LustreError: 334324:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9658.887991] Lustre: Skipped 2 previous similar messages [ 9658.932523] LustreError: 334324:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9659.438703] Lustre: server umount lustre-OST0000 complete [ 9663.467963] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9663.481881] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9669.097199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9669.099207] Lustre: Skipped 2 previous similar messages [ 9669.936446] Lustre: server umount lustre-MDT0000 complete [ 9673.810936] LustreError: 332711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789138749 with bad export cookie 13052812020825271278 [ 9673.813729] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9673.829961] LustreError: 332711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9674.479519] Lustre: server umount lustre-MDT0001 complete [ 9685.669289] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 9688.712792] Key type lgssc unregistered [ 9689.060124] LNet: 337362:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9689.069133] LNetError: 337362:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9689.096621] LNet: Removed LNI 192.168.206.122@tcp [ 9689.900358] Key type .llcrypt unregistered [ 9689.903167] Key type ._llcrypt unregistered [ 9767.889194] Key type ._llcrypt registered [ 9767.892817] Key type .llcrypt registered [ 9768.047293] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [ 9785.742574] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 9787.292191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9787.421104] alg: No test for adler32 (adler32-zlib) [ 9788.644586] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 9788.970653] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 9790.695438] Key type lgssc registered [ 9792.046746] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9827.901130] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9827.927330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9829.234139] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9829.262927] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9829.335773] Lustre: lustre-MDT0000: new disk, initializing [ 9829.430610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9829.449373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9834.383250] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9845.277932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9845.362670] Lustre: 342306:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9845.425967] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9845.434321] Lustre: Skipped 1 previous similar message [ 9845.505674] Lustre: lustre-MDT0001: new disk, initializing [ 9845.612276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9845.638216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9845.644279] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9850.177945] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9860.260273] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9867.309195] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9874.622792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9875.039764] Lustre: lustre-OST0000: new disk, initializing [ 9875.042919] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9875.054526] Lustre: 344278:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9875.163960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9882.206918] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9882.714192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9882.730612] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9882.847179] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9892.239690] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9897.214193] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9897.402075] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9902.205734] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9902.538990] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9908.704981] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9908.714681] LustreError: Skipped 1 previous similar message [ 9908.714687] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9908.724828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9908.762971] Lustre: Skipped 1 previous similar message [ 9910.679329] Lustre: server umount lustre-OST0000 complete [ 9917.408864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9917.425994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9917.444308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9917.452240] Lustre: Skipped 1 previous similar message [ 9918.944955] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9918.947306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9918.991698] Lustre: Skipped 1 previous similar message [ 9924.066723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9928.163211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9928.176887] Lustre: Skipped 2 previous similar messages [ 9929.183249] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9929.383497] Lustre: server umount lustre-MDT0000 complete [ 9933.130861] LustreError: 342301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789139009 with bad export cookie 5817489334835523964 [ 9933.132168] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9933.139628] LustreError: 342301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9933.723423] Lustre: server umount lustre-MDT0001 complete [ 9942.160709] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 11:03:36 (1789139016) [ 9950.460347] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 11:03:44 (1789139024) [ 9959.153879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9959.714947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9964.237488] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9965.026645] LustreError: 347034:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9965.073986] LustreError: 347034:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9969.124324] LustreError: 347035:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9974.069634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9974.244380] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9974.505460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9979.514779] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9987.200686] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9993.727319] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10000.334421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10000.690661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10007.303854] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10015.340410] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10026.839336] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10028.946452] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10030.743608] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10032.460377] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10034.274506] Lustre: Failing over lustre-MDT0000 [10034.504379] Lustre: server umount lustre-MDT0000 complete [10035.695347] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10035.701742] LustreError: 348565:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10035.712515] Lustre: Skipped 2 previous similar messages [10035.726073] LustreError: 348565:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10037.856701] LustreError: 347031:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10037.870184] LustreError: 347031:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10042.948186] LustreError: 347028:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10042.984642] LustreError: 347028:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10051.041439] Lustre: 340534:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789139111/real 1789139111] req@0000000055fcc6e0 x1876048073529472/t0(0) o400->MGC192.168.206.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789139127 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10051.043268] LustreError: 347035:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10051.074227] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10051.098832] LustreError: 347035:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [10052.880813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10061.280576] LustreError: 340530:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000096775f14 x1876048073534080/t0(0) o250->MGC192.168.206.122@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10061.506351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10061.550533] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10063.418714] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10065.310919] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10066.934157] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10066.989819] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10067.036650] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10073.966248] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10075.345258] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10078.813441] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10081.019424] Lustre: Failing over lustre-MDT0000 [10082.279532] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10082.280939] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10082.292172] LustreError: 348034:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10082.292186] LustreError: 348034:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 8 previous similar messages [10082.337664] Lustre: Skipped 2 previous similar messages [10083.225894] Lustre: server umount lustre-MDT0000 complete [10098.655285] Lustre: 340533:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789139158/real 1789139158] req@000000003995856d x1876048073548800/t0(0) o400->MGC192.168.206.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789139174 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10098.703337] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10103.856452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10109.413756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10109.507958] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10110.533023] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10113.751237] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10114.554000] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10114.558760] Lustre: Skipped 2 previous similar messages [10114.590858] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10114.637983] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10122.800768] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10124.457765] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10135.008786] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10135.012031] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10135.012920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10135.038040] Lustre: Skipped 2 previous similar messages [10136.836080] Lustre: server umount lustre-MDT0000 complete [10140.022330] LustreError: 347014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789139216 with bad export cookie 5817489334835526680 [10140.035058] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10140.035205] LustreError: 347014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10140.129539] LustreError: 347028:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10140.142155] LustreError: 347028:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 26 previous similar messages [10140.430823] Lustre: server umount lustre-MDT0001 complete [10146.231260] Lustre: server umount lustre-OST0000 complete [10162.862682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10169.607370] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10179.399752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10179.817448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10179.829456] Lustre: Skipped 1 previous similar message [10183.988063] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10190.282204] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10195.004358] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10202.112122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10207.928946] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10208.563781] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10214.217745] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10218.985536] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10218.988506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10219.006524] Lustre: Skipped 3 previous similar messages [10219.010041] Lustre: Skipped 2 previous similar messages [10225.094205] Lustre: server umount lustre-OST0000 complete [10234.667126] Lustre: server umount lustre-MDT0000 complete [10238.496370] LustreError: 353385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789139314 with bad export cookie 5817489334835527429 [10238.509061] LustreError: 353385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10238.513100] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10238.790432] Lustre: server umount lustre-MDT0001 complete [10249.641835] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [10253.511161] Key type lgssc unregistered [10253.957377] LNet: 356937:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10253.962743] LNetError: 356937:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10255.029579] LNet: Removed LNI 192.168.206.122@tcp [10256.094550] Key type .llcrypt unregistered [10256.097927] Key type ._llcrypt unregistered [10284.084476] Key type ._llcrypt registered [10284.087611] Key type .llcrypt registered [10284.242143] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [10285.197141] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10285.236635] alg: No test for adler32 (adler32-zlib) [10286.432752] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [10286.724940] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [10288.495432] Key type lgssc registered [10290.163636] Lustre: Echo OBD driver; http://www.lustre.org/ [10297.929286] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10297.948804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10299.453835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10303.551633] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10309.609939] LustreError: 358455:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10309.630910] LustreError: 358455:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10312.947051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10313.485303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10319.208537] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10327.832630] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10333.758132] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10341.480303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10341.839306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10344.966829] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10348.316442] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10356.207525] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10368.487280] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10368.489909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10368.505464] Lustre: Skipped 1 previous similar message [10368.515301] Lustre: Skipped 2 previous similar messages [10373.256454] Lustre: server umount lustre-MDT0000 complete [10373.606776] LustreError: 358449:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10373.631499] LustreError: 358449:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10376.727448] LustreError: 358436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789139452 with bad export cookie 12310555651980673985 [10376.735709] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10376.738246] LustreError: 358436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10377.076172] Lustre: server umount lustre-MDT0001 complete [10390.302549] Lustre: server umount lustre-OST0000 complete [10402.101212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10402.628695] LustreError: 362166:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10402.724978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10406.243527] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10407.908212] LustreError: 362167:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10412.003869] LustreError: 362166:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10412.758760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10413.100048] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10417.169995] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10423.959725] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10429.022302] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10435.189892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10435.550721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10441.724417] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10447.856698] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10448.636819] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10463.202792] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10463.207554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10463.223624] Lustre: Skipped 3 previous similar messages [10463.239299] Lustre: Skipped 2 previous similar messages [10465.994710] Lustre: server umount lustre-MDT0000 complete [10468.320272] LustreError: 362166:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10468.347149] LustreError: 362166:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10469.209597] LustreError: 362146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789139545 with bad export cookie 12310555651980675098 [10469.212899] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10469.216329] LustreError: 362146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10469.475734] Lustre: server umount lustre-MDT0001 complete [10484.142207] Lustre: server umount lustre-OST0000 complete [10498.757076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10499.428924] LustreError: 366077:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10499.514677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10504.127609] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10513.867589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10519.546852] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10527.115412] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10532.392585] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10538.917441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10539.216448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10539.220786] Lustre: Skipped 1 previous similar message [10544.385420] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10544.811330] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10550.494773] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10554.849298] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10554.850304] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10554.858640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10554.867825] Lustre: Skipped 1 previous similar message [10554.876164] Lustre: Skipped 1 previous similar message [10559.972562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10559.977901] Lustre: Skipped 1 previous similar message [10560.706675] Lustre: server umount lustre-OST0000 complete [10565.090861] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10565.093812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10565.099404] Lustre: Skipped 1 previous similar message [10570.390324] Lustre: server umount lustre-MDT0000 complete [10573.268519] LustreError: 366812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789139649 with bad export cookie 12310555651980676120 [10573.269468] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10573.284505] LustreError: 366812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10573.620947] Lustre: server umount lustre-MDT0001 complete [10582.276246] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [10584.838792] Key type lgssc unregistered [10585.112680] LNet: 369614:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10585.120826] LNetError: 369614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10585.149877] LNet: Removed LNI 192.168.206.122@tcp [10585.916779] Key type .llcrypt unregistered [10585.918882] Key type ._llcrypt unregistered [10608.724238] Key type ._llcrypt registered [10608.725829] Key type .llcrypt registered [10608.857932] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [10609.717841] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10609.798695] alg: No test for adler32 (adler32-zlib) [10610.863764] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [10611.108559] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [10612.743184] Key type lgssc registered [10613.722929] Lustre: Echo OBD driver; http://www.lustre.org/ [10620.511938] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10620.528631] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10620.533961] LustreError: 371033:0:(osd_handler.c:8517:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10620.548360] LustreError: 371033:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10620.553345] LustreError: 371033:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10620.559506] LustreError: 371033:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10620.574484] LustreError: 371033:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10626.805816] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10626.813958] LustreError: 371429:0:(osd_handler.c:8517:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10626.835630] LustreError: 371429:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10626.840769] LustreError: 371429:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10626.845893] LustreError: 371429:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10626.850965] LustreError: 371429:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10642.855990] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [10645.704597] Key type lgssc unregistered [10645.997392] LNet: 372436:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10646.007521] LNetError: 372436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10646.067071] LNet: Removed LNI 192.168.206.122@tcp [10646.910545] Key type .llcrypt unregistered [10646.914068] Key type ._llcrypt unregistered [10664.746442] Key type ._llcrypt registered [10664.748393] Key type .llcrypt registered [10664.816669] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [10681.810618] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [10683.245959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10683.600813] alg: No test for adler32 (adler32-zlib) [10685.166785] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [10685.553555] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [10687.335841] Key type lgssc registered [10689.216995] Lustre: Echo OBD driver; http://www.lustre.org/ [10722.682971] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10722.736616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10724.064442] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10724.094670] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10724.162642] Lustre: lustre-MDT0000: new disk, initializing [10724.231149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10724.262499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10728.386442] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10737.820364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10737.938194] Lustre: 375599:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10738.005488] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10738.009991] Lustre: Skipped 1 previous similar message [10738.084115] Lustre: lustre-MDT0001: new disk, initializing [10738.205529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10738.244314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10738.250507] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10741.840547] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10750.345619] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10756.773734] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10763.514597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10763.735517] Lustre: lustre-OST0000: new disk, initializing [10763.742440] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10763.756146] Lustre: 377566:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10763.831122] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10769.322977] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10772.095160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10772.113549] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10772.301033] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10779.082993] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10783.774906] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10784.067322] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10788.784358] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10789.067908] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10792.929357] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10792.930983] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10792.940660] LustreError: Skipped 1 previous similar message [10792.965028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10792.984371] Lustre: Skipped 1 previous similar message [10793.007863] Lustre: Skipped 1 previous similar message [10797.525402] Lustre: server umount lustre-OST0000 complete [10803.169327] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10803.175386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10803.190142] Lustre: Skipped 1 previous similar message [10803.198890] Lustre: Skipped 1 previous similar message [10807.572448] Lustre: server umount lustre-MDT0000 complete [10811.820865] LustreError: 375590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789139887 with bad export cookie 10676867615752013259 [10811.825075] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10811.839277] LustreError: 375590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10812.181313] Lustre: server umount lustre-MDT0001 complete [10819.323733] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10820.585516] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 11:18:15 (1789139895) [10826.406211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10826.793685] LustreError: 380025:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10826.815400] LustreError: 380025:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10826.869644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10830.943077] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10832.354533] LustreError: 380026:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10836.457928] LustreError: 380025:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10838.655672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10839.140935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10843.687898] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10852.294847] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10858.443341] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10866.272945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10866.685237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10874.288985] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10882.570414] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10890.257397] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10890.521140] Lustre: lustre-OST0001: new disk, initializing [10890.524472] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10890.533721] Lustre: 382860:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10890.672981] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10897.743818] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10899.543955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10899.557320] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10899.651040] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10907.110347] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10914.895405] Lustre: lustre-OST0001: Not available for connect from 192.168.206.22@tcp (stopping) [10915.317568] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10915.319079] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10915.338551] LustreError: Skipped 1 previous similar message [10915.358463] Lustre: Skipped 1 previous similar message [10918.509841] Lustre: server umount lustre-OST0001 complete [10920.003387] LustreError: 381848:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10925.544808] LustreError: 381848:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10925.566973] LustreError: 381848:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10935.266656] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10935.281447] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10935.298700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10935.306627] Lustre: Skipped 2 previous similar messages [10935.801344] LustreError: 382551:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10935.822659] LustreError: 382551:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10940.911314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10940.919123] Lustre: Skipped 2 previous similar messages [10941.274395] Lustre: server umount lustre-OST0000 complete [10946.024734] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10946.051109] Lustre: Skipped 1 previous similar message [10951.137655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10951.142764] Lustre: Skipped 3 previous similar messages [10959.331090] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10959.521819] Lustre: server umount lustre-MDT0000 complete [10961.378148] LustreError: 380025:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10961.396792] LustreError: 380025:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10962.856911] LustreError: 380007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789140038 with bad export cookie 10676867615752014134 [10962.865193] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10962.879131] LustreError: 380007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10977.250368] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10977.442151] Lustre: server umount lustre-MDT0001 complete [10991.845542] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [10995.333586] Key type lgssc unregistered [10995.631398] LNet: 384852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10995.635231] LNetError: 384852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10995.659825] LNet: Removed LNI 192.168.206.122@tcp [10996.983159] Key type .llcrypt unregistered [10996.988837] Key type ._llcrypt unregistered [11046.193203] Key type ._llcrypt registered [11046.195943] Key type .llcrypt registered [11046.300920] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [11047.384392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11047.400079] alg: No test for adler32 (adler32-zlib) [11048.454310] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [11048.698949] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [11050.368489] Key type lgssc registered [11051.334766] Lustre: Echo OBD driver; http://www.lustre.org/ [11057.630773] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11057.655994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11058.925496] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11058.953462] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11059.149093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11063.261606] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [11071.530762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11071.655492] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11071.679800] Lustre: 387650:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11071.974575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11077.125284] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [11084.757636] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11090.191145] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11092.450349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11092.462492] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11092.478646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11094.499409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11094.499634] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11098.557974] Lustre: server umount lustre-MDT0000 complete [11102.056237] LustreError: 387641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789140178 with bad export cookie 9103770552563031701 [11102.062137] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11102.067918] LustreError: 387641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11102.636766] Lustre: server umount lustre-MDT0001 complete [11111.401249] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 11:23:06 (1789140186) [11117.642990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11118.020624] LustreError: 390163:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11118.032926] LustreError: 390163:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11118.070691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11121.627765] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [11123.178212] LustreError: 390164:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11128.291048] LustreError: 390163:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11129.227517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11129.606654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11133.592705] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [11140.282856] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11145.341891] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11151.246468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11151.472704] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11151.633679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11156.905446] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [11162.737872] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11168.622970] Lustre: Permanently deactivating lustre-OST0000 [11168.638611] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11175.465609] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11176.715604] LustreError: 393148:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11176.723864] LustreError: 393148:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11176.732752] LustreError: 393148:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11179.084376] LustreError: 393196:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11179.090652] LustreError: 393196:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11186.811275] Lustre: server umount lustre-OST0000 complete [11189.162833] LustreError: 393446:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11189.170038] LustreError: 393446:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11191.264625] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11191.269116] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11191.293834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11191.301767] Lustre: Skipped 1 previous similar message [11194.336225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11194.337674] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11199.458309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11199.467132] Lustre: Skipped 2 previous similar messages [11200.530462] Lustre: server umount lustre-MDT0000 complete [11203.611083] LustreError: 390143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789140279 with bad export cookie 9103770552563032282 [11203.625453] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11213.926559] Lustre: server umount lustre-MDT0001 complete [11221.757353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11226.541116] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [11229.164884] LustreError: 394604:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11235.901821] Lustre: 394944:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.122@tcp [11238.585770] Lustre: server umount MGS complete [11251.685488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11252.051703] LustreError: 395669:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11252.073673] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11252.080797] Lustre: Skipped 1 previous similar message [11252.161388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11255.956659] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [11257.312548] LustreError: 395670:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11263.364074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11263.575636] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11267.432584] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [11274.424228] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11280.700157] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11282.674538] Lustre: Permanently reactivating lustre-OST0000 [11282.704231] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11282.722177] Lustre: Skipped 2 previous similar messages [11308.513779] LustreError: 397389:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11316.957196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11317.355528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11317.360468] Lustre: Skipped 1 previous similar message [11322.904938] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [11329.094392] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11334.116136] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11334.120629] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11334.143842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11334.153405] Lustre: Skipped 1 previous similar message [11335.386476] Lustre: server umount lustre-OST0000 complete [11345.072798] Lustre: server umount lustre-MDT0000 complete [11348.440682] LustreError: 397242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789140424 with bad export cookie 9103770552563033325 [11348.446188] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11348.448952] LustreError: 397242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11348.774276] Lustre: server umount lustre-MDT0001 complete [11359.958919] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [11362.436342] Key type lgssc unregistered [11362.786548] LNet: 399987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11362.799345] LNetError: 399987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11362.822825] LNet: Removed LNI 192.168.206.122@tcp [11363.709505] Key type .llcrypt unregistered [11363.711505] Key type ._llcrypt unregistered [11379.801519] Key type ._llcrypt registered [11379.802931] Key type .llcrypt registered [11379.899068] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [11397.254151] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [11398.691970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11398.719448] alg: No test for adler32 (adler32-zlib) [11399.885654] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [11400.192252] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [11401.891142] Key type lgssc registered [11402.926638] Lustre: Echo OBD driver; http://www.lustre.org/ [11436.677689] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11436.708368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11437.913261] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11437.982525] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11438.103976] Lustre: lustre-MDT0000: new disk, initializing [11438.218617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11438.296570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11442.529507] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [11451.161875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11451.244864] Lustre: 403148:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11451.303166] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11451.308316] Lustre: Skipped 1 previous similar message [11451.393783] Lustre: lustre-MDT0001: new disk, initializing [11451.492103] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11451.531421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11451.549877] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11455.243793] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [11462.491446] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11467.424606] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11473.414703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11473.675691] Lustre: lustre-OST0000: new disk, initializing [11473.679301] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11473.683513] Lustre: 405116:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11473.756016] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11479.488842] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [11482.738457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11482.749423] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11482.868744] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11489.491100] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11494.813989] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11494.997994] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11499.984492] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11500.202602] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11503.589720] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11503.596206] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11503.627101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11503.638616] Lustre: Skipped 1 previous similar message [11508.711193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11508.731537] Lustre: Skipped 1 previous similar message [11513.825913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11513.847536] Lustre: Skipped 1 previous similar message [11516.383152] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11516.720686] Lustre: server umount lustre-OST0000 complete [11523.040725] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11523.055736] LustreError: Skipped 1 previous similar message [11523.066922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11523.083751] Lustre: Skipped 1 previous similar message [11523.106410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11524.069294] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11527.700239] Lustre: server umount lustre-MDT0000 complete [11531.616750] LustreError: 403137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789140607 with bad export cookie 5556193947186709904 [11531.621766] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11531.626769] LustreError: 403137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11531.974759] Lustre: server umount lustre-MDT0001 complete [11539.429316] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11540.996098] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 11:30:15 (1789140615) [11547.053630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11547.487546] LustreError: 407574:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11547.500076] LustreError: 407574:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11547.556963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11551.682742] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [11552.751650] LustreError: 407575:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11557.869505] LustreError: 407574:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11559.371088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11559.652865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11563.906283] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [11569.623168] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11574.917648] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11581.109895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11581.498258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11587.599163] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [11593.400862] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11595.048559] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11597.315625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11602.914422] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11602.930573] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11602.954898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11602.963848] Lustre: Skipped 3 previous similar messages [11608.480368] Lustre: server umount lustre-OST0000 complete [11613.161618] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11613.162622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11613.171281] Lustre: Skipped 2 previous similar messages [11613.184120] Lustre: Skipped 3 previous similar messages [11618.115622] Lustre: server umount lustre-MDT0000 complete [11618.282605] LustreError: 407570:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11618.307564] LustreError: 407570:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11621.607046] LustreError: 407556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789140697 with bad export cookie 5556193947186710786 [11621.615378] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11621.636215] LustreError: 407556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11621.926742] Lustre: server umount lustre-MDT0001 complete [11634.158350] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [11636.838888] Key type lgssc unregistered [11637.076733] LNet: 411300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11637.085943] LNetError: 411300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11637.104919] LNet: Removed LNI 192.168.206.122@tcp [11638.179584] Key type .llcrypt unregistered [11638.181864] Key type ._llcrypt unregistered [11661.356725] Key type ._llcrypt registered [11661.359498] Key type .llcrypt registered [11661.442810] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [11661.688265] Key type .llcrypt unregistered [11661.694748] Key type ._llcrypt unregistered [11681.286917] Key type ._llcrypt registered [11681.289636] Key type .llcrypt registered [11681.409799] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [11682.294952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11682.506616] alg: No test for adler32 (adler32-zlib) [11683.517844] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [11683.705545] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [11685.399172] Key type lgssc registered [11686.594989] Lustre: Echo OBD driver; http://www.lustre.org/ [11694.130122] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11694.163189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11695.690586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11699.957249] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [11705.825832] LustreError: 413518:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11705.874849] LustreError: 413518:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11710.541167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11710.965797] LustreError: 413517:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11711.207939] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11716.585283] LustreError: 413514:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11716.597417] LustreError: 413514:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11719.934699] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [11720.685063] LustreError: 413517:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11720.725722] LustreError: 413517:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11720.732883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11725.797329] LustreError: 413518:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11725.810550] LustreError: 413518:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11728.103289] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11734.833654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11735.124770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11740.571954] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [11750.368736] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11750.390508] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11750.402346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11750.423923] Lustre: Skipped 1 previous similar message [11750.437226] Lustre: Skipped 1 previous similar message [11750.785152] Lustre: server umount lustre-OST0000 complete [11755.493737] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11755.494873] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11755.505928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11755.514823] Lustre: Skipped 1 previous similar message [11756.834522] Lustre: server umount lustre-MDT0000 complete [11760.281940] LustreError: 413497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789140836 with bad export cookie 15159585169359424020 [11760.293662] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11760.302949] LustreError: 413497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11760.629553] Lustre: server umount lustre-MDT0001 complete [11770.858643] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [11773.216087] Key type lgssc unregistered [11773.453959] LNet: 416670:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11773.471017] LNetError: 416670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11773.496213] LNet: Removed LNI 192.168.206.122@tcp [11774.342123] Key type .llcrypt unregistered [11774.344809] Key type ._llcrypt unregistered [11797.123574] Key type ._llcrypt registered [11797.125334] Key type .llcrypt registered [11797.267676] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [11798.059224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11798.101157] alg: No test for adler32 (adler32-zlib) [11799.446773] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [11799.890239] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [11801.583258] Key type lgssc registered [11802.880222] Lustre: Echo OBD driver; http://www.lustre.org/ [11811.334830] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11811.362880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11826.918637] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11832.031335] LustreError: 418067:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.122@tcp: failed processing log, type 4: rc = -110 [11858.655198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11864.815809] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [11871.111228] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11877.126512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11877.539897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11877.641933] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11881.596730] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [11882.992392] LustreError: 418093:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11883.008464] LustreError: 418093:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11888.098068] LustreError: 418092:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11890.194860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11890.820278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11896.408460] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [11904.008758] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11910.211415] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11916.256183] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11916.260140] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11916.268497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11918.182941] Lustre: server umount lustre-OST0000 complete [11923.943356] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [11923.949587] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11923.964570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11925.985529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11925.987083] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11931.104782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11931.117734] Lustre: Skipped 2 previous similar messages [11932.244238] Lustre: server umount lustre-MDT0000 complete [11935.206305] LustreError: 419110:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11935.226935] LustreError: 419110:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11936.130167] LustreError: 418074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789141012 with bad export cookie 13913885826094759090 [11936.137369] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11936.143685] LustreError: 418074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11936.415698] Lustre: server umount lustre-MDT0001 complete [11945.830567] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [11948.245355] Key type lgssc unregistered [11948.477525] LNet: 421641:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11948.494631] LNetError: 421641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11948.505278] LNet: Removed LNI 192.168.206.122@tcp [11949.561352] Key type .llcrypt unregistered [11949.565053] Key type ._llcrypt unregistered [11978.894160] Key type ._llcrypt registered [11978.901254] Key type .llcrypt registered [11979.011826] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [11980.346519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11980.397416] alg: No test for adler32 (adler32-zlib) [11981.519696] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [11981.813923] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [11983.551378] Key type lgssc registered [11985.155123] Lustre: Echo OBD driver; http://www.lustre.org/ [11995.692132] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11995.742298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11997.226478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12003.683830] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [12007.402181] LustreError: 423059:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12007.468106] LustreError: 423059:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12012.512685] LustreError: 423060:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12014.241483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12014.483397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12017.979303] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [12025.471403] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12033.787562] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12043.633112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12044.111146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12050.558181] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [12057.912115] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12067.694675] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12072.934278] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12072.940197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12072.955793] Lustre: Skipped 1 previous similar message [12075.103064] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.22@tcp (stopping) [12075.109218] Lustre: Skipped 2 previous similar messages [12078.052319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12078.056834] Lustre: Skipped 2 previous similar messages [12078.822873] Lustre: server umount lustre-MDT0000 complete [12080.184626] LustreError: 423056:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12080.976413] LustreError: 423039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.22@tcp arrived at 1789141156 with bad export cookie 12007767197842303675 [12082.826903] LustreError: 423040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789141158 with bad export cookie 12007767197842303017 [12082.838811] LustreError: 423040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12082.844177] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12083.171263] LustreError: 423055:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12083.178241] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12083.186114] LustreError: 423055:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12083.205441] Lustre: Skipped 1 previous similar message [12083.214621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12087.200997] LustreError: 423788:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12087.212343] LustreError: 423788:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12089.120517] Lustre: server umount lustre-OST0000 complete [12095.891576] Lustre: server umount lustre-MDT0001 complete [12107.384143] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [12109.891396] Key type lgssc unregistered [12110.166639] LNet: 426773:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12110.177114] LNetError: 426773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12110.198892] LNet: Removed LNI 192.168.206.122@tcp [12111.113915] Key type .llcrypt unregistered [12111.117344] Key type ._llcrypt unregistered [12135.118271] Key type ._llcrypt registered [12135.121578] Key type .llcrypt registered [12135.335862] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [12136.771426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12136.813359] alg: No test for adler32 (adler32-zlib) [12137.870774] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [12138.111793] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [12139.799962] Key type lgssc registered [12140.844965] Lustre: Echo OBD driver; http://www.lustre.org/ [12150.182060] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12150.200442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12151.922395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12156.452674] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [12161.510724] LustreError: 428192:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12161.524906] LustreError: 428192:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12165.379091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12165.686989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12170.918736] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [12179.276165] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12185.267792] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12193.220951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12193.807380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12196.915277] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12201.036784] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [12209.163349] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12215.272298] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12215.287205] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12215.299610] Lustre: Skipped 1 previous similar message [12215.891749] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.22@tcp (stopping) [12219.460373] Lustre: server umount lustre-MDT0001 complete [12220.398120] LustreError: 428189:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12220.407169] LustreError: 428189:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12225.512834] LustreError: 428192:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12225.551787] LustreError: 428192:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12227.553069] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12227.564036] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12227.587182] Lustre: Skipped 1 previous similar message [12227.597665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12230.625518] LustreError: 428193:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12230.628894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12232.061675] Lustre: server umount lustre-OST0000 complete [12235.746123] LustreError: 428192:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12236.649633] Lustre: server umount lustre-MDT0000 complete [12251.673094] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [12254.674453] Key type lgssc unregistered [12255.049766] LNet: 431880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12255.059883] LNetError: 431880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12255.081071] LNet: Removed LNI 192.168.206.122@tcp [12256.013992] Key type .llcrypt unregistered [12256.016339] Key type ._llcrypt unregistered [12280.337352] Key type ._llcrypt registered [12280.340526] Key type .llcrypt registered [12280.447083] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [12295.120578] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [12296.165687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12296.268576] alg: No test for adler32 (adler32-zlib) [12297.320667] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [12297.579530] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [12299.255198] Key type lgssc registered [12300.205973] Lustre: Echo OBD driver; http://www.lustre.org/ [12340.452083] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [12356.865519] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12356.905373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12358.188286] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12358.216438] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12358.308800] Lustre: lustre-MDT0000: new disk, initializing [12358.371810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12358.383360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12362.755650] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [12377.402188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12377.758647] Lustre: lustre-OST0000: new disk, initializing [12377.763653] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12377.770996] Lustre: Skipped 1 previous similar message [12377.775639] Lustre: 436915:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12377.840935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12384.717929] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [12385.417786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12385.434375] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12385.484569] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12394.437265] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12411.081415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12411.226119] Lustre: 435898:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12411.334398] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12411.450469] Lustre: lustre-MDT0001: new disk, initializing [12411.650227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12411.724914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12411.742399] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12416.478679] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [12425.751353] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12425.940053] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12430.180859] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12430.374961] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12436.462842] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12436.467616] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12436.485267] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12436.510624] Lustre: Skipped 1 previous similar message [12437.579301] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.22@tcp (stopping) [12437.585258] Lustre: Skipped 1 previous similar message [12441.575615] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12441.584722] Lustre: Skipped 1 previous similar message [12446.692210] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12446.704160] Lustre: Skipped 2 previous similar messages [12449.247227] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12449.589283] Lustre: server umount lustre-MDT0001 complete [12456.934887] LustreError: 435910:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12456.962992] LustreError: 435910:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [12462.066685] LustreError: 435904:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12462.090078] LustreError: 435904:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12467.171557] LustreError: 438518:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12467.193154] LustreError: 438518:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12472.290339] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12472.298493] LustreError: 435909:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12472.335104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12472.357766] Lustre: Skipped 1 previous similar message [12477.414391] LustreError: 435910:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12482.532909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12482.546452] Lustre: Skipped 1 previous similar message [12483.039352] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12483.144733] Lustre: server umount lustre-OST0000 complete [12487.650141] LustreError: 435909:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12487.705836] LustreError: 435909:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12495.254032] Lustre: server umount lustre-MDT0000 complete [12517.383645] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [12521.606303] Key type lgssc unregistered [12522.043655] LNet: 440679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12522.055533] LNetError: 440679:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12522.078246] LNet: Removed LNI 192.168.206.122@tcp [12523.151616] Key type .llcrypt unregistered [12523.153687] Key type ._llcrypt unregistered [12568.852596] Key type ._llcrypt registered [12568.859857] Key type .llcrypt registered [12569.013559] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [12569.429526] Key type .llcrypt unregistered [12569.431267] Key type ._llcrypt unregistered [12587.551168] Key type ._llcrypt registered [12587.553451] Key type .llcrypt registered [12587.645472] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [12589.438730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12589.476541] alg: No test for adler32 (adler32-zlib) [12590.538947] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [12590.795809] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [12592.495321] Key type lgssc registered [12594.254660] Lustre: Echo OBD driver; http://www.lustre.org/ [12623.701357] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12623.726709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12625.001894] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12625.035576] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12625.115634] Lustre: lustre-MDT0000: new disk, initializing [12625.265039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12625.305242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12630.062460] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [12641.141651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12641.280442] Lustre: 444061:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12641.355695] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12641.365028] Lustre: Skipped 1 previous similar message [12641.472299] Lustre: lustre-MDT0001: new disk, initializing [12641.605477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12641.677875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12641.701050] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12646.334511] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [12656.600358] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12663.240896] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12670.929862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12671.219272] Lustre: lustre-OST0000: new disk, initializing [12671.223908] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12671.229770] Lustre: 446029:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12671.318104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12676.710118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12676.727491] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12676.858348] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12677.678836] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [12688.187578] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12697.580152] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12697.586288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12697.606420] Lustre: Skipped 1 previous similar message [12697.625655] Lustre: Skipped 2 previous similar messages [12702.545815] Lustre: server umount lustre-MDT0000 complete [12706.836245] LustreError: 444053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789141782 with bad export cookie 12190028245955818745 [12706.853781] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12706.857028] LustreError: 444053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12707.633669] Lustre: server umount lustre-MDT0001 complete [12721.262851] Lustre: server umount lustre-OST0000 complete [12733.100601] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 11:50:07 (1789141807) [12741.963610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12742.618738] LustreError: 448058:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12742.654324] LustreError: 448058:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [12742.749200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12747.744720] LustreError: 448059:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12748.220628] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [12752.866900] LustreError: 448058:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12757.177772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12757.433058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12762.233263] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [12770.774815] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12776.325265] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12796.589246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12796.783764] Lustre: Found index 0 for lustre-OST0000, updating log [12797.088988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12804.526499] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [12813.491885] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12814.759629] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12823.011284] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12823.022441] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12823.039674] Lustre: Skipped 1 previous similar message [12823.049462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12824.101255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12824.110816] Lustre: Skipped 1 previous similar message [12830.182244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12830.191802] Lustre: Skipped 1 previous similar message [12834.272134] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12834.470554] Lustre: server umount lustre-OST0000 complete [12850.337046] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12850.342827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12850.343952] Lustre: Skipped 2 previous similar messages [12850.352926] Lustre: Skipped 2 previous similar messages [12854.048604] LNet: 1 local NIs in recovery (showing 1): 192.168.206.122@tcp [12856.702507] Lustre: server umount lustre-MDT0000 complete [12860.899014] LustreError: 448059:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12860.907970] LustreError: 448059:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12860.927539] LustreError: 448794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789141936 with bad export cookie 12190028245955820131 [12860.930147] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12860.947118] LustreError: 448794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12861.293822] Lustre: server umount lustre-MDT0001 complete [12871.165102] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 11:52:25 (1789141945) [13492.777563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13493.356102] LustreError: 454483:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13493.460647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13498.267653] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [13498.853970] LustreError: 454484:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13503.969815] LustreError: 454483:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13506.839387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13507.269073] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13511.897373] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [13520.108934] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13526.296258] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13545.247863] LNet: 1 local NIs in recovery (showing 1): 192.168.206.122@tcp [13546.622257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13546.990305] Lustre: Found index 0 for lustre-OST0000, updating log [13547.408963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13554.160416] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [13561.920806] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13565.921136] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13565.937060] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13565.965758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13565.974439] Lustre: Skipped 2 previous similar messages [13570.482231] Lustre: server umount lustre-OST0000 complete [13574.112332] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13574.121187] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13574.129728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13575.143746] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13576.236124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13576.241980] Lustre: Skipped 3 previous similar messages [13580.324375] LustreError: 454478:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13580.337250] LustreError: 454478:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13580.466487] Lustre: server umount lustre-MDT0000 complete [13584.417403] LustreError: 454483:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13584.437693] LustreError: 454483:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 8 previous similar messages [13584.578130] LustreError: 456339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789142660 with bad export cookie 12190028245955821223 [13584.600670] LustreError: 456339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13584.604159] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13585.098632] Lustre: server umount lustre-MDT0001 complete [13605.663331] LNet: 1 local NIs in recovery (showing 1): 192.168.206.122@tcp [13617.129875] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [13627.514957] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [13668.045807] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 12:05:42 (1789142742) [13686.797692] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [13689.493651] Key type lgssc unregistered [13689.756072] LNet: 461106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13689.766945] LNetError: 461106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13809.582214] LNet: Removed LNI 192.168.206.122@tcp [13810.421874] Key type .llcrypt unregistered [13810.428318] Key type ._llcrypt unregistered [13826.823575] Key type ._llcrypt registered [13826.826099] Key type .llcrypt registered [13826.964710] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [13828.166836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13828.282120] alg: No test for adler32 (adler32-zlib) [13829.307385] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [13829.524212] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [13831.225123] Key type lgssc registered [13832.442820] Lustre: Echo OBD driver; http://www.lustre.org/ [13836.971243] LNet: Added LNI 192.168.206.122@tcp100 [8/256/0/180] [13837.743429] LNet: Added LNI 192.168.206.122@tcp101 [8/256/0/180] [13838.554727] LNet: Added LNI 192.168.206.122@tcp102 [8/256/0/180] [13839.413348] LNet: Added LNI 192.168.206.122@tcp103 [8/256/0/180] [13840.314062] LNet: Added LNI 192.168.206.122@tcp104 [8/256/0/180] [13841.256663] LNet: Added LNI 192.168.206.122@tcp105 [8/256/0/180] [13842.215598] LNet: Added LNI 192.168.206.122@tcp106 [8/256/0/180] [13843.045649] LNet: Added LNI 192.168.206.122@tcp107 [8/256/0/180] [13843.914305] LNet: Added LNI 192.168.206.122@tcp108 [8/256/0/180] [13844.740468] LNet: Added LNI 192.168.206.122@tcp109 [8/256/0/180] [13845.590106] LNet: Added LNI 192.168.206.122@tcp110 [8/256/0/180] [13846.681110] LNet: Added LNI 192.168.206.122@tcp111 [8/256/0/180] [13847.614564] LNet: Added LNI 192.168.206.122@tcp112 [8/256/0/180] [13848.708837] LNet: Added LNI 192.168.206.122@tcp113 [8/256/0/180] [13849.819352] LNet: Added LNI 192.168.206.122@tcp114 [8/256/0/180] [13850.764439] LNet: Added LNI 192.168.206.122@tcp115 [8/256/0/180] [13851.889731] LNet: Added LNI 192.168.206.122@tcp116 [8/256/0/180] [13852.772736] LNet: Added LNI 192.168.206.122@tcp117 [8/256/0/180] [13853.612631] LNet: Added LNI 192.168.206.122@tcp118 [8/256/0/180] [13854.737820] LNet: Added LNI 192.168.206.122@tcp119 [8/256/0/180] [13855.615245] LNet: Added LNI 192.168.206.122@tcp120 [8/256/0/180] [13856.610873] LNet: Added LNI 192.168.206.122@tcp121 [8/256/0/180] [13857.580286] LNet: Added LNI 192.168.206.122@tcp122 [8/256/0/180] [13858.658471] LNet: Added LNI 192.168.206.122@tcp123 [8/256/0/180] [13859.830986] LNet: Added LNI 192.168.206.122@tcp124 [8/256/0/180] [13860.773407] LNet: Added LNI 192.168.206.122@tcp125 [8/256/0/180] [13861.818884] LNet: Added LNI 192.168.206.122@tcp126 [8/256/0/180] [13862.896613] LNet: Added LNI 192.168.206.122@tcp127 [8/256/0/180] [13863.967048] LNet: Added LNI 192.168.206.122@tcp128 [8/256/0/180] [13865.159945] LNet: Added LNI 192.168.206.122@tcp129 [8/256/0/180] [13866.054646] LNet: Added LNI 192.168.206.122@tcp130 [8/256/0/180] [13867.014454] LNet: Added LNI 192.168.206.122@tcp131 [8/256/0/180] [13868.053596] LNet: Added LNI 192.168.206.122@tcp132 [8/256/0/180] [13869.074697] LNet: Added LNI 192.168.206.122@tcp133 [8/256/0/180] [13870.139370] LNet: Added LNI 192.168.206.122@tcp134 [8/256/0/180] [13871.143199] LNet: Added LNI 192.168.206.122@tcp135 [8/256/0/180] [13879.233628] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13879.274672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13880.405466] Lustre: 464140:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.122@tcp: can't add NID 192.168.206.122@tcp130: rc = -75 [13880.501681] LNetError: 461599:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.122 [13880.510508] LNetError: Protocol error connecting to 192.168.206.122@tcp115 at host 192.168.206.122:988: is it running a compatible version of Lustre? [13880.540162] LNetError: Skipped 7 previous similar messages [13880.622343] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13880.650762] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13880.736327] LNet: 8 local NIs in recovery (showing 8): 192.168.206.122@tcp122, 192.168.206.122@tcp120, 192.168.206.122@tcp123, 192.168.206.122@tcp124, 192.168.206.122@tcp126, 192.168.206.122@tcp127, 192.168.206.122@tcp128, 192.168.206.122@tcp125 [13880.755423] Lustre: lustre-MDT0000: new disk, initializing [13880.926246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13880.955437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13881.825609] LNetError: 464163:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.122 [13881.839447] LNetError: 464163:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 14 previous similar messages [13881.855625] LNetError: Protocol error connecting to 192.168.206.122@tcp122 at host 192.168.206.122:988: is it running a compatible version of Lustre? [13881.867349] LNetError: Skipped 5 previous similar messages [13883.879601] LNetError: 461596:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.122 [13883.893232] LNetError: 461596:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [13883.908517] LNetError: Protocol error connecting to 192.168.206.122@tcp122 at host 192.168.206.122:988: is it running a compatible version of Lustre? [13883.934850] LNetError: Skipped 1 previous similar message [13886.202678] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [13887.976944] LNetError: 461599:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.122 [13888.000404] LNetError: Protocol error connecting to 192.168.206.122@tcp122 at host 192.168.206.122:988: is it running a compatible version of Lustre? [13898.738123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13898.948299] Lustre: 464157:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13899.012976] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13899.016243] Lustre: Skipped 1 previous similar message [13899.100333] Lustre: lustre-MDT0001: new disk, initializing [13899.133405] LNetError: 464161:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.122 [13899.147464] LNetError: Protocol error connecting to 192.168.206.122@tcp130 at host 192.168.206.122:988: is it running a compatible version of Lustre? [13899.148808] Lustre: 464959:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.122@tcp: can't add NID 192.168.206.122@tcp130: rc = -75 [13899.181261] Lustre: 464959:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13899.379750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13899.452569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13899.465937] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13904.044621] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [13909.064774] Lustre: 465384:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.122@tcp: can't add NID 192.168.206.122@tcp130: rc = -75 [13909.072371] Lustre: 465384:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13910.432996] LNetError: 464163:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.122 [13910.457799] LNetError: 464163:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 21 previous similar messages [13910.470239] LNetError: Protocol error connecting to 192.168.206.122@tcp117 at host 192.168.206.122:988: is it running a compatible version of Lustre? [13910.486152] LNetError: Skipped 21 previous similar messages [13914.359757] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13920.877280] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13926.828420] LNetError: 461599:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.122 [13926.844390] LNetError: 461599:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 16 previous similar messages [13926.862857] LNetError: Protocol error connecting to 192.168.206.122@tcp128 at host 192.168.206.122:988: is it running a compatible version of Lustre? [13926.875249] LNetError: Skipped 16 previous similar messages [13929.514506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13929.866938] Lustre: lustre-OST0000: new disk, initializing [13929.877707] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13929.886453] Lustre: 466134:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13930.024215] Lustre: 466139:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.122@tcp: can't add NID 192.168.206.122@tcp130: rc = -75 [13930.031141] Lustre: 466139:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13930.074463] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13936.727817] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [13938.270441] Lustre: 466564:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.122@tcp: can't add NID 192.168.206.122@tcp130: rc = -75 [13938.287295] Lustre: 466564:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13938.365727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13938.379786] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13938.548462] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13940.260528] LNet: 6 local NIs in recovery (showing 6): 192.168.206.122@tcp130, 192.168.206.122@tcp131, 192.168.206.122@tcp133, 192.168.206.122@tcp134, 192.168.206.122@tcp135, 192.168.206.122@tcp122 [13946.595157] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13959.155888] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13959.164595] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13959.175815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13959.177035] LustreError: Skipped 1 previous similar message [13959.205635] Lustre: Skipped 1 previous similar message [13959.590183] LNetError: 464160:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.122 [13959.600491] LNetError: 464160:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 31 previous similar messages [13959.609536] LNetError: Protocol error connecting to 192.168.206.122@tcp132 at host 192.168.206.122:988: is it running a compatible version of Lustre? [13959.618944] LNetError: Skipped 31 previous similar messages [13960.519593] Lustre: server umount lustre-OST0000 complete [13965.797847] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13965.810175] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13965.834771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13965.848459] Lustre: Skipped 1 previous similar message [13969.380769] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13969.386627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13969.416390] Lustre: Skipped 1 previous similar message [13974.497625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13974.506600] Lustre: Skipped 1 previous similar message [13979.103130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13979.412640] Lustre: server umount lustre-MDT0000 complete [13984.268789] LustreError: 464150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789143060 with bad export cookie 12218898346687199071 [13984.290039] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13984.299386] LustreError: 464150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13984.874507] Lustre: server umount lustre-MDT0001 complete [13997.121463] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [13999.984689] Key type lgssc unregistered [14000.373305] LNet: 468053:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14000.407693] LNetError: 468053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14001.439798] LNet: Removed LNI 192.168.206.122@tcp [14001.467479] LNetError: 468053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14001.487533] LNet: Removed LNI 192.168.206.122@tcp100 [14002.527467] LNet: Removed LNI 192.168.206.122@tcp101 [14002.531783] LNetError: 468053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14002.542448] LNetError: 468053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14002.558121] LNet: Removed LNI 192.168.206.122@tcp102 [14003.615156] LNet: Removed LNI 192.168.206.122@tcp103 [14004.639155] LNet: Removed LNI 192.168.206.122@tcp104 [14004.641160] LNetError: 468053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14004.657067] LNetError: 468053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14004.662645] LNet: Removed LNI 192.168.206.122@tcp105 [14005.727472] LNet: Removed LNI 192.168.206.122@tcp106 [14006.751427] LNet: Removed LNI 192.168.206.122@tcp107 [14007.775189] LNet: Removed LNI 192.168.206.122@tcp108 [14008.799329] LNet: Removed LNI 192.168.206.122@tcp109 [14008.818038] LNetError: 468053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14008.828242] LNetError: 468053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14008.836543] LNet: Removed LNI 192.168.206.122@tcp110 [14009.887356] LNet: Removed LNI 192.168.206.122@tcp111 [14010.911159] LNet: Removed LNI 192.168.206.122@tcp112 [14011.936313] LNet: Removed LNI 192.168.206.122@tcp113 [14012.959264] LNet: Removed LNI 192.168.206.122@tcp114 [14013.983302] LNet: Removed LNI 192.168.206.122@tcp115 [14015.007126] LNet: Removed LNI 192.168.206.122@tcp116 [14016.031331] LNet: Removed LNI 192.168.206.122@tcp117 [14017.055371] LNet: Removed LNI 192.168.206.122@tcp118 [14017.066509] LNetError: 468053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14017.075261] LNetError: 468053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14017.090873] LNet: Removed LNI 192.168.206.122@tcp119 [14018.144956] LNet: Removed LNI 192.168.206.122@tcp120 [14019.167226] LNet: Removed LNI 192.168.206.122@tcp121 [14019.169924] LNet: Removed LNI 192.168.206.122@tcp122 [14020.191634] LNet: Removed LNI 192.168.206.122@tcp123 [14021.215283] LNet: Removed LNI 192.168.206.122@tcp124 [14022.239179] LNet: Removed LNI 192.168.206.122@tcp125 [14023.263296] LNet: Removed LNI 192.168.206.122@tcp126 [14024.287261] LNet: Removed LNI 192.168.206.122@tcp127 [14025.311236] LNet: Removed LNI 192.168.206.122@tcp128 [14026.335207] LNet: Removed LNI 192.168.206.122@tcp129 [14026.337597] LNet: Removed LNI 192.168.206.122@tcp130 [14026.346813] LNet: Removed LNI 192.168.206.122@tcp131 [14027.423330] LNet: Removed LNI 192.168.206.122@tcp132 [14027.425421] LNet: Removed LNI 192.168.206.122@tcp133 [14027.431658] LNet: Removed LNI 192.168.206.122@tcp134 [14027.447759] LNet: Removed LNI 192.168.206.122@tcp135 [14028.528215] Key type .llcrypt unregistered [14028.532638] Key type ._llcrypt unregistered [14053.775815] Key type ._llcrypt registered [14053.781931] Key type .llcrypt registered [14053.891029] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [14070.299802] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [14071.851537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14071.887291] alg: No test for adler32 (adler32-zlib) [14072.904619] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [14073.142272] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [14074.815388] Key type lgssc registered [14076.017285] Lustre: Echo OBD driver; http://www.lustre.org/ [14113.319669] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 12:13:07 (1789143187) [14131.895509] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [14134.969845] Key type lgssc unregistered [14135.358304] LNet: 472146:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14135.371754] LNetError: 472146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14135.387851] LNet: Removed LNI 192.168.206.122@tcp [14136.355123] Key type .llcrypt unregistered [14136.358908] Key type ._llcrypt unregistered [14153.065222] Key type ._llcrypt registered [14153.068634] Key type .llcrypt registered [14153.220879] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [14154.940335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14154.976806] alg: No test for adler32 (adler32-zlib) [14156.095201] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [14156.377288] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [14158.079240] Key type lgssc registered [14159.601340] Lustre: Echo OBD driver; http://www.lustre.org/ [14169.920876] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14169.976954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14171.278977] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14171.301416] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14171.374365] Lustre: lustre-MDT0000: new disk, initializing [14171.485319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14171.502295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14176.476572] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [14189.076416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14189.215823] Lustre: 473362:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14189.280780] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14189.285169] Lustre: Skipped 1 previous similar message [14189.358333] Lustre: lustre-MDT0001: new disk, initializing [14189.463596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14189.509203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14189.523079] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14194.902382] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [14205.272659] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14212.270751] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14219.880705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14220.236554] Lustre: lustre-OST0000: new disk, initializing [14220.239930] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14220.248790] Lustre: 475333:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14220.340640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14226.919780] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [14228.104960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14228.125548] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14228.324988] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14238.011850] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14242.797938] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing lnet_if_list [14248.168457] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14248.211417] LNet: Added LNI 192.168.206.122@tcp10 [8/256/0/180] [14253.922445] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14253.978157] LNet: Added LNI 192.168.206.122@tcp11 [8/256/0/180] [14258.318549] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14258.339604] LNet: Added LNI 192.168.206.122@tcp12 [8/256/0/180] [14262.591536] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14262.627530] LNet: Added LNI 192.168.206.122@tcp13 [8/256/0/180] [14266.997609] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14267.053501] LNet: Added LNI 192.168.206.122@tcp14 [8/256/0/180] [14271.473272] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14271.504317] LNet: Added LNI 192.168.206.122@tcp15 [8/256/0/180] [14276.189177] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14276.238524] LNet: Added LNI 192.168.206.122@tcp16 [8/256/0/180] [14281.430836] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14281.458446] LNet: Added LNI 192.168.206.122@tcp17 [8/256/0/180] [14287.064839] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14287.128893] LNet: Added LNI 192.168.206.122@tcp18 [8/256/0/180] [14292.654750] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14292.677520] LNet: Added LNI 192.168.206.122@tcp19 [8/256/0/180] [14297.594699] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14297.615785] LNet: Added LNI 192.168.206.122@tcp20 [8/256/0/180] [14302.733949] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14302.767292] LNet: Added LNI 192.168.206.122@tcp21 [8/256/0/180] [14307.861925] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14307.894844] LNet: Added LNI 192.168.206.122@tcp22 [8/256/0/180] [14313.238433] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14313.295255] LNet: Added LNI 192.168.206.122@tcp23 [8/256/0/180] [14319.645648] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14319.700586] LNet: Added LNI 192.168.206.122@tcp24 [8/256/0/180] [14325.480656] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14325.528557] LNet: Added LNI 192.168.206.122@tcp25 [8/256/0/180] [14331.548477] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14331.579588] LNet: Added LNI 192.168.206.122@tcp26 [8/256/0/180] [14337.484908] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14337.512980] LNet: Added LNI 192.168.206.122@tcp27 [8/256/0/180] [14343.995293] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14344.024146] LNet: Added LNI 192.168.206.122@tcp28 [8/256/0/180] [14349.941628] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14349.986787] LNet: Added LNI 192.168.206.122@tcp29 [8/256/0/180] [14355.669927] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14355.707044] LNet: Added LNI 192.168.206.122@tcp30 [8/256/0/180] [14362.260624] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14362.294478] LNet: Added LNI 192.168.206.122@tcp31 [8/256/0/180] [14369.093659] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14369.122279] LNet: Added LNI 192.168.206.122@tcp32 [8/256/0/180] [14375.308975] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14375.343967] LNet: Added LNI 192.168.206.122@tcp33 [8/256/0/180] [14383.666609] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14383.706115] LNet: Added LNI 192.168.206.122@tcp34 [8/256/0/180] [14391.354319] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14391.403688] LNet: Added LNI 192.168.206.122@tcp35 [8/256/0/180] [14399.596830] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14399.635271] LNet: Added LNI 192.168.206.122@tcp36 [8/256/0/180] [14407.484978] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14407.511238] LNet: Added LNI 192.168.206.122@tcp37 [8/256/0/180] [14417.651573] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14417.705468] LNet: Added LNI 192.168.206.122@tcp38 [8/256/0/180] [14428.355834] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14428.382974] LNet: Added LNI 192.168.206.122@tcp39 [8/256/0/180] [14437.577620] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14437.621515] LNet: Added LNI 192.168.206.122@tcp40 [8/256/0/180] [14445.725973] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14445.792865] LNet: Added LNI 192.168.206.122@tcp41 [8/256/0/180] [14455.185580] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14455.211311] LNet: Added LNI 192.168.206.122@tcp42 [8/256/0/180] [14464.731147] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14464.756557] LNet: Added LNI 192.168.206.122@tcp43 [8/256/0/180] [14474.375351] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14474.451081] LNet: Added LNI 192.168.206.122@tcp44 [8/256/0/180] [14485.613618] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14485.657782] LNet: Added LNI 192.168.206.122@tcp45 [8/256/0/180] [14495.495147] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14495.536256] LNet: Added LNI 192.168.206.122@tcp46 [8/256/0/180] [14505.130489] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14505.172816] LNet: Added LNI 192.168.206.122@tcp47 [8/256/0/180] [14515.026652] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14515.075514] LNet: Added LNI 192.168.206.122@tcp48 [8/256/0/180] [14522.875160] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14522.899045] LNet: Added LNI 192.168.206.122@tcp49 [8/256/0/180] [14531.172114] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14531.200402] LNet: Added LNI 192.168.206.122@tcp50 [8/256/0/180] [14551.020369] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14551.027803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14551.047927] Lustre: Skipped 2 previous similar messages [14551.072455] Lustre: Skipped 2 previous similar messages [14554.787042] Lustre: server umount lustre-MDT0000 complete [14558.413378] LustreError: 475343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789143634 with bad export cookie 12601138701560722813 [14558.414685] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14558.423149] LustreError: 475343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14558.700397] Lustre: server umount lustre-MDT0001 complete [14576.668840] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [14577.638496] Lustre: 472634:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789143637/real 1789143637] req@00000000bbd6b011 x1876052653813760/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789143653 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14577.660404] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14579.743226] Lustre: 472634:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789143639/real 1789143639] req@00000000391e69d6 x1876052653813888/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789143655 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14582.752041] Lustre: 472635:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789143642/real 1789143642] req@0000000050179e77 x1876052653814016/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789143658 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14584.870133] Lustre: 472635:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789143644/real 1789143644] req@00000000752bd9e9 x1876052653814272/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789143660 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14585.762816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14589.919422] Lustre: 472633:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789143649/real 1789143649] req@00000000bef5f5c2 x1876052653814400/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789143665 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14603.960125] LustreError: 475325:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14603.986243] LustreError: 475325:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [14604.080661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14606.112587] LustreError: 475326:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14606.125519] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14606.144481] LustreError: 475326:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [14607.146195] LustreError: 475325:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14611.352614] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [14612.464402] LustreError: 475326:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14617.571320] LustreError: 475325:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14622.143875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14622.725328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14623.747578] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14629.055861] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [14637.377489] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14642.999930] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14664.682813] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14664.691192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14664.702026] Lustre: Skipped 1 previous similar message [14664.718529] Lustre: Skipped 1 previous similar message [14669.805876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14669.822000] Lustre: Skipped 1 previous similar message [14670.629226] Lustre: server umount lustre-OST0000 complete [14674.913878] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14674.916402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14674.940107] Lustre: Skipped 1 previous similar message [14674.961509] Lustre: Skipped 1 previous similar message [14680.034091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14680.046272] Lustre: Skipped 1 previous similar message [14681.015570] Lustre: server umount lustre-MDT0000 complete [14684.820126] LustreError: 475343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789143760 with bad export cookie 12601138701560725417 [14684.824458] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14684.831114] LustreError: 475343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14685.129315] Lustre: server umount lustre-MDT0001 complete [14696.120946] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [14699.061924] Key type lgssc unregistered [14699.588952] LNet: 502391:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14699.596768] LNetError: 502391:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14700.639317] LNet: Removed LNI 192.168.206.122@tcp [14700.644227] LNetError: 502391:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14700.652161] LNet: Removed LNI 192.168.206.122@tcp10 [14700.656245] LNet: Removed LNI 192.168.206.122@tcp11 [14700.659220] LNet: Removed LNI 192.168.206.122@tcp12 [14700.662592] LNet: Removed LNI 192.168.206.122@tcp13 [14700.666155] LNet: Removed LNI 192.168.206.122@tcp14 [14700.669136] LNet: Removed LNI 192.168.206.122@tcp15 [14700.671624] LNet: Removed LNI 192.168.206.122@tcp16 [14700.673771] LNet: Removed LNI 192.168.206.122@tcp17 [14700.676609] LNet: Removed LNI 192.168.206.122@tcp18 [14700.679654] LNet: Removed LNI 192.168.206.122@tcp19 [14700.682589] LNet: Removed LNI 192.168.206.122@tcp20 [14700.685217] LNet: Removed LNI 192.168.206.122@tcp21 [14700.688496] LNet: Removed LNI 192.168.206.122@tcp22 [14700.691488] LNet: Removed LNI 192.168.206.122@tcp23 [14700.694601] LNet: Removed LNI 192.168.206.122@tcp24 [14700.697749] LNet: Removed LNI 192.168.206.122@tcp25 [14700.700732] LNet: Removed LNI 192.168.206.122@tcp26 [14700.703584] LNet: Removed LNI 192.168.206.122@tcp27 [14700.707134] LNet: Removed LNI 192.168.206.122@tcp28 [14700.709158] LNet: Removed LNI 192.168.206.122@tcp29 [14700.712592] LNet: Removed LNI 192.168.206.122@tcp30 [14700.715602] LNet: Removed LNI 192.168.206.122@tcp31 [14700.718574] LNet: Removed LNI 192.168.206.122@tcp32 [14700.721758] LNet: Removed LNI 192.168.206.122@tcp33 [14700.724353] LNet: Removed LNI 192.168.206.122@tcp34 [14700.727334] LNet: Removed LNI 192.168.206.122@tcp35 [14700.729330] LNet: Removed LNI 192.168.206.122@tcp36 [14700.733204] LNet: Removed LNI 192.168.206.122@tcp37 [14700.736351] LNet: Removed LNI 192.168.206.122@tcp38 [14700.739180] LNet: Removed LNI 192.168.206.122@tcp39 [14700.742366] LNet: Removed LNI 192.168.206.122@tcp40 [14700.745394] LNet: Removed LNI 192.168.206.122@tcp41 [14700.748419] LNet: Removed LNI 192.168.206.122@tcp42 [14700.750816] LNet: Removed LNI 192.168.206.122@tcp43 [14700.753809] LNet: Removed LNI 192.168.206.122@tcp44 [14700.756449] LNet: Removed LNI 192.168.206.122@tcp45 [14700.759416] LNet: Removed LNI 192.168.206.122@tcp46 [14700.762502] LNet: Removed LNI 192.168.206.122@tcp47 [14700.764583] LNet: Removed LNI 192.168.206.122@tcp48 [14700.769612] LNet: Removed LNI 192.168.206.122@tcp49 [14700.794545] LNet: Removed LNI 192.168.206.122@tcp50 [14701.481989] Key type .llcrypt unregistered [14701.484917] Key type ._llcrypt unregistered [14723.068603] Key type ._llcrypt registered [14723.071966] Key type .llcrypt registered [14723.215540] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [14739.375867] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [14740.440865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14740.507335] alg: No test for adler32 (adler32-zlib) [14741.619318] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [14741.879564] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [14743.583243] Key type lgssc registered [14744.666217] Lustre: Echo OBD driver; http://www.lustre.org/ [14780.549673] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14780.635493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14781.927040] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14781.976563] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14782.041261] Lustre: lustre-MDT0000: new disk, initializing [14782.092746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14782.106689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14786.599184] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [14796.986330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14797.081913] Lustre: 505586:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14797.125771] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14797.129563] Lustre: Skipped 1 previous similar message [14797.214927] Lustre: lustre-MDT0001: new disk, initializing [14797.357632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14797.399452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14797.428660] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14801.740524] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [14810.862687] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14816.538962] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14824.261077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14824.446358] Lustre: *** cfs_fail_loc=90f, val=0*** [14824.548157] Lustre: lustre-OST0000: new disk, initializing [14824.555932] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14824.568431] Lustre: 507604:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14824.681303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14830.514126] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [14831.711752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14831.720693] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14831.835405] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14839.240566] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14847.468832] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14847.482743] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14847.488078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14847.511738] Lustre: Skipped 1 previous similar message [14847.544932] LustreError: Skipped 1 previous similar message [14851.841585] Lustre: server umount lustre-OST0000 complete [14857.700419] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14857.707543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14857.716883] Lustre: Skipped 1 previous similar message [14857.728992] Lustre: Skipped 1 previous similar message [14861.749713] Lustre: server umount lustre-MDT0000 complete [14865.502323] LustreError: 505580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789143941 with bad export cookie 11580906744986677250 [14865.518867] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14865.520572] LustreError: 505580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14865.913921] Lustre: server umount lustre-MDT0001 complete [14877.085425] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [14879.981099] Key type lgssc unregistered [14880.227649] LNet: 509512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14880.233516] LNetError: 509512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14880.247724] LNet: Removed LNI 192.168.206.122@tcp [14881.121352] Key type .llcrypt unregistered [14881.123071] Key type ._llcrypt unregistered [14900.540732] Key type ._llcrypt registered [14900.548284] Key type .llcrypt registered [14900.676881] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [14901.698462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14901.752701] alg: No test for adler32 (adler32-zlib) [14902.950131] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [14903.270558] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [14905.031210] Key type lgssc registered [14906.181759] Lustre: Echo OBD driver; http://www.lustre.org/ [14913.995846] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14914.010751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14915.486638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14920.157444] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [14925.795164] LustreError: 510794:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14925.834254] LustreError: 510794:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [14929.576733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14930.133198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14935.245240] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [14942.801900] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14948.405272] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14954.761982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14955.123895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14961.273607] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [14969.258868] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14979.880666] loop0: detected capacity change from 0 to 204800000 [14980.056604] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14980.077859] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14980.089074] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14986.980723] loop0: detected capacity change from 0 to 204800000 [14987.209285] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14987.229618] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14987.252521] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14993.580531] loop0: detected capacity change from 0 to 204800000 [14999.656215] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14999.819650] Lustre: fs15246-OST0000: new disk, initializing [14999.823547] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14999.829798] Lustre: 514388:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14999.887083] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15006.314545] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [15016.633157] loop1: detected capacity change from 0 to 204800000 [15022.197380] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15022.297529] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15022.336928] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15022.417929] Lustre: fs15246-MDT0000: new disk, initializing [15022.553857] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15022.578386] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15022.650364] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15023.713516] Lustre: 515511:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000067064ace x1876053436971776/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789144115 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15026.695460] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [15032.814820] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15032.839297] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15032.871105] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15052.266882] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15052.294587] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15056.903777] Lustre: server umount fs15246-MDT0000 complete [15057.387508] LustreError: 510789:0:(ldlm_lib.c:1228:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15062.513524] LustreError: 510789:0:(ldlm_lib.c:1228:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15067.338474] Lustre: server umount fs15246-OST0000 complete [15082.981544] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15082.990775] Lustre: Skipped 1 previous similar message [15082.993286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15083.005930] Lustre: Skipped 1 previous similar message [15087.128738] Lustre: server umount lustre-OST0000 complete [15093.217203] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15093.246351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15093.267239] Lustre: Skipped 1 previous similar message [15098.338673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15098.350047] Lustre: Skipped 1 previous similar message [15099.517062] Lustre: server umount lustre-MDT0000 complete [15103.461663] LustreError: 512036:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15103.508857] LustreError: 512036:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [15104.629871] LustreError: 510777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789144180 with bad export cookie 2271407443512801805 [15104.639116] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15104.645953] LustreError: 510777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15105.373984] Lustre: server umount lustre-MDT0001 complete [15122.347832] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [15126.162306] Key type lgssc unregistered [15126.405372] LNet: 518614:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15126.413759] LNetError: 518614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15126.435478] LNet: Removed LNI 192.168.206.122@tcp [15127.434163] Key type .llcrypt unregistered [15127.439075] Key type ._llcrypt unregistered [15147.212747] Key type ._llcrypt registered [15147.216092] Key type .llcrypt registered [15147.330582] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [15148.536341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15148.809488] alg: No test for adler32 (adler32-zlib) [15149.858707] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [15150.135933] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [15151.824294] Key type lgssc registered [15152.878383] Lustre: Echo OBD driver; http://www.lustre.org/ [15195.588477] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [15207.113620] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [15244.026865] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15244.047431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15245.260646] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15245.322312] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15245.425057] Lustre: lustre-MDT0000: new disk, initializing [15245.562235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15245.587720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15250.746929] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [15261.502249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15261.591325] Lustre: 523082:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15261.644916] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15261.648160] Lustre: Skipped 1 previous similar message [15261.710739] Lustre: lustre-MDT0001: new disk, initializing [15261.810599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15261.852470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15261.860769] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15265.830417] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [15276.353949] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15281.956790] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15288.164742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15288.385180] Lustre: lustre-OST0000: new disk, initializing [15288.389869] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15288.395497] Lustre: 525053:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15288.515847] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15294.051425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15294.058229] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15294.171617] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15294.646156] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [15303.556693] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15308.174800] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15308.371782] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15312.863626] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15313.120627] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15320.036469] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15320.042087] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15320.055288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15320.072657] Lustre: Skipped 1 previous similar message [15320.084882] Lustre: Skipped 1 previous similar message [15321.528786] Lustre: server umount lustre-OST0000 complete [15325.153273] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15325.160954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15325.167901] Lustre: Skipped 1 previous similar message [15330.276024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15330.285969] Lustre: Skipped 1 previous similar message [15331.349364] Lustre: server umount lustre-MDT0000 complete [15335.211049] LustreError: 523073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789144411 with bad export cookie 12505571363728874559 [15335.225960] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15335.227943] LustreError: 523073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15335.542669] Lustre: server umount lustre-MDT0001 complete [15366.996463] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [15378.314821] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [15411.058918] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15110 sec ========= 12:34:45 (1789144485) [15413.038528] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:34:47 (1789144487) === [15416.029873] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:34:50 (1789144490) === [15453.961683] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [15456.860809] Key type lgssc unregistered [15457.074229] LNet: 531701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15457.082214] LNetError: 531701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15458.154992] LNet: Removed LNI 192.168.206.122@tcp [15458.931524] Key type .llcrypt unregistered [15458.933575] Key type ._llcrypt unregistered