[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 478571208 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002246] x2apic enabled [ 0.003000] Switched APIC routing to physical x2apic. [ 0.003010] kvm-guest: setup PV IPIs [ 0.005397] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007009] pid_max: default: 32768 minimum: 301 [ 0.008140] LSM: Security Framework initializing [ 0.009044] Yama: becoming mindful. [ 0.010028] SELinux: Initializing. [ 0.011043] *** VALIDATE selinux *** [ 0.019347] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024415] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026112] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028087] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029167] *** VALIDATE tmpfs *** [ 0.031327] *** VALIDATE proc *** [ 0.032308] *** VALIDATE cgroup *** [ 0.033009] *** VALIDATE cgroup2 *** [ 0.035007] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036185] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037017] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039032] Spectre V2 : User space: Vulnerable [ 0.040011] Speculative Store Bypass: Vulnerable [ 0.043370] debug: unmapping init [mem 0xffffffffa3059000-0xffffffffa3060fff] [ 0.045470] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047664] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048030] ... version: 2 [ 0.049017] ... bit width: 48 [ 0.050015] ... generic registers: 4 [ 0.051015] ... value mask: 0000ffffffffffff [ 0.052020] ... max period: 00007fffffffffff [ 0.053015] ... fixed-purpose events: 3 [ 0.054011] ... event mask: 000000070000000f [ 0.055355] rcu: Hierarchical SRCU implementation. [ 0.057713] smp: Bringing up secondary CPUs ... [ 0.058744] x86: Booting SMP configuration: [ 0.059031] .... node #0, CPUs: #1 #2 #3 [ 0.066018] smp: Brought up 1 node, 4 CPUs [ 0.068012] smpboot: Max logical packages: 1 [ 0.069011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.126050] node 0 deferred pages initialised in 55ms [ 0.129330] devtmpfs: initialized [ 0.130155] x86/mm: Memory block size: 128MB [ 0.133040] gcov: version magic: 0x41383552 [ 0.135361] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.139082] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.141303] pinctrl core: initialized pinctrl subsystem [ 0.142147] [ 0.142576] ************************************************************* [ 0.144010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.146009] ** ** [ 0.147007] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.149008] ** ** [ 0.150007] ** This means that this kernel is built to expose internal ** [ 0.151010] ** IOMMU data structures, which may compromise security on ** [ 0.153007] ** your system. ** [ 0.154008] ** ** [ 0.156008] ** If you see this message and you are not debugging the ** [ 0.157006] ** kernel, report this immediately to your vendor! ** [ 0.159008] ** ** [ 0.160008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161007] ************************************************************* [ 0.163654] NET: Registered protocol family 16 [ 0.164360] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.166035] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.167034] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.170008] cpuidle: using governor menu [ 0.171103] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.172322] PCI: Using configuration type 1 for base access [ 0.174140] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.180109] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.181024] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.182302] cryptd: max_cpu_qlen set to 1000 [ 0.184254] ACPI: Added _OSI(Module Device) [ 0.185012] ACPI: Added _OSI(Processor Device) [ 0.185910] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.187012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.190068] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.194343] ACPI: Interpreter enabled [ 0.195041] ACPI: PM: (supports S0 S3 S4 S5) [ 0.195837] ACPI: Using IOAPIC for interrupt routing [ 0.196089] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.198511] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.208022] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.210028] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.211010] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.213202] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.218580] acpiphp: Slot [2] registered [ 0.220093] acpiphp: Slot [5] registered [ 0.221097] acpiphp: Slot [6] registered [ 0.222105] acpiphp: Slot [7] registered [ 0.223098] acpiphp: Slot [8] registered [ 0.224087] acpiphp: Slot [9] registered [ 0.225112] acpiphp: Slot [10] registered [ 0.226083] acpiphp: Slot [3] registered [ 0.227083] acpiphp: Slot [4] registered [ 0.228089] acpiphp: Slot [11] registered [ 0.229079] acpiphp: Slot [12] registered [ 0.230072] acpiphp: Slot [13] registered [ 0.231141] acpiphp: Slot [14] registered [ 0.232127] acpiphp: Slot [15] registered [ 0.233033] acpiphp: Slot [16] registered [ 0.234116] acpiphp: Slot [17] registered [ 0.235070] acpiphp: Slot [18] registered [ 0.236075] acpiphp: Slot [19] registered [ 0.237067] acpiphp: Slot [20] registered [ 0.238078] acpiphp: Slot [21] registered [ 0.239055] acpiphp: Slot [22] registered [ 0.240076] acpiphp: Slot [23] registered [ 0.241123] acpiphp: Slot [24] registered [ 0.242091] acpiphp: Slot [25] registered [ 0.244112] acpiphp: Slot [26] registered [ 0.245088] acpiphp: Slot [27] registered [ 0.246071] acpiphp: Slot [28] registered [ 0.247042] acpiphp: Slot [29] registered [ 0.248151] acpiphp: Slot [30] registered [ 0.249090] acpiphp: Slot [31] registered [ 0.250036] PCI host bridge to bus 0000:00 [ 0.250943] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.252030] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.254018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.255020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.257029] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.259029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.260168] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.262903] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.265275] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.274019] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.278479] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.280021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.281011] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.283012] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.284516] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.286740] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.289055] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.290790] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.295013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.305027] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.310917] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.315736] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.322013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.327020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.343023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.354779] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.359018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.364019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.380019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.388264] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.396077] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.402024] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.420015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.434335] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.441018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.447018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.462045] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.473545] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.478014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.487014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.499020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.507073] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.516016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.522021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.545019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.554126] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.557386] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.559379] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.561201] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.562219] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.566066] iommu: Default domain type: Passthrough [ 0.567543] SCSI subsystem initialized [ 0.569125] ACPI: bus type USB registered [ 0.570082] usbcore: registered new interface driver usbfs [ 0.571062] usbcore: registered new interface driver hub [ 0.572000] usbcore: registered new device driver usb [ 0.572000] pps_core: LinuxPPS API ver. 1 registered [ 0.573009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.574076] PTP clock support registered [ 0.575123] EDAC MC: Ver: 3.0.0 [ 0.576465] PCI: Using ACPI for IRQ routing [ 0.577695] NetLabel: Initializing [ 0.578008] NetLabel: domain hash size = 128 [ 0.578973] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.579061] NetLabel: unlabeled traffic allowed by default [ 0.580112] vgaarb: loaded [ 0.581231] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.582007] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.586993] clocksource: Switched to clocksource kvm-clock [ 0.699166] VFS: Disk quotas dquot_6.6.0 [ 0.700241] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.701782] *** VALIDATE ramfs *** [ 0.702553] *** VALIDATE hugetlbfs *** [ 0.703518] pnp: PnP ACPI init [ 0.705241] pnp: PnP ACPI: found 6 devices [ 0.718382] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.720412] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.721723] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.723067] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.724532] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.725989] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.727781] NET: Registered protocol family 2 [ 0.730049] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.735549] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.739173] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.743772] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.746313] TCP: Hash tables configured (established 65536 bind 65536) [ 0.748235] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.750313] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.752199] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.754237] NET: Registered protocol family 1 [ 0.755913] RPC: Registered named UNIX socket transport module. [ 0.757259] RPC: Registered udp transport module. [ 0.758225] RPC: Registered tcp transport module. [ 0.759404] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.761024] NET: Registered protocol family 44 [ 0.762181] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.763532] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.764810] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.766281] PCI: CLS 0 bytes, default 64 [ 0.767910] Unpacking initramfs... [ 2.135567] debug: unmapping init [mem 0xffff89887cc54000-0xffff89887ffbffff] [ 2.138434] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.140024] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.141979] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.609411] Initialise system trusted keyrings [ 2.610732] Key type blacklist registered [ 2.612180] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.621134] zbud: loaded [ 2.623753] *** VALIDATE nfs *** [ 2.624772] *** VALIDATE nfs4 *** [ 2.626124] pstore: using deflate compression [ 2.629123] Platform Keyring initialized [ 2.724405] NET: Registered protocol family 38 [ 2.725874] Key type asymmetric registered [ 2.727202] Asymmetric key parser 'x509' registered [ 2.728459] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.730856] io scheduler mq-deadline registered [ 2.732103] io scheduler kyber registered [ 2.733324] io scheduler bfq registered [ 2.734830] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.736788] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.738583] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.740369] ACPI: Power Button [PWRF] [ 2.743701] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.748747] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.758457] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.766469] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.777541] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.803484] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.830252] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.834656] Non-volatile memory driver v1.3 [ 2.836229] Linux agpgart interface v0.103 [ 2.864165] virtio_blk virtio1: [vda] 149960 512-byte logical blocks (76.8 MB/73.2 MiB) [ 2.867127] vda: detected capacity change from 0 to 76779520 [ 2.881834] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.885475] vdb: detected capacity change from 0 to 1073741824 [ 2.909234] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.913235] vdc: detected capacity change from 0 to 2621440000 [ 2.934875] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.940137] vdd: detected capacity change from 0 to 2621440000 [ 2.960893] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.963720] vde: detected capacity change from 0 to 4294967296 [ 2.983479] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.986650] vdf: detected capacity change from 0 to 4294967296 [ 2.994688] libphy: Fixed MDIO Bus: probed [ 3.001039] usbcore: registered new interface driver usbserial_generic [ 3.003968] usbserial: USB Serial support registered for generic [ 3.006334] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.010539] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.012391] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.015026] mousedev: PS/2 mouse device common for all mice [ 3.018810] rtc_cmos 00:05: RTC can wake from S4 [ 3.021613] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.023075] rtc_cmos 00:05: registered as rtc0 [ 3.027912] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.031159] intel_pstate: CPU model not supported [ 3.035123] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.035648] hid: raw HID events driver (C) Jiri Kosina [ 3.040051] usbcore: registered new interface driver usbhid [ 3.040101] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.041964] usbhid: USB HID core driver [ 3.042138] drop_monitor: Initializing network drop monitor service [ 3.048787] Initializing XFRM netlink socket [ 3.051027] NET: Registered protocol family 10 [ 3.054881] Segment Routing with IPv6 [ 3.056309] NET: Registered protocol family 17 [ 3.058477] mpls_gso: MPLS GSO support [ 3.063891] RAS: Correctable Errors collector initialized. [ 3.065544] AVX version of gcm_enc/dec engaged. [ 3.066678] AES CTR mode by8 optimization enabled [ 3.147930] sched_clock: Marking stable (3147900749, 0)->(3989874325, -841973576) [ 3.151497] registered taskstats version 1 [ 3.153587] Loading compiled-in X.509 certificates [ 3.155729] zswap: loaded using pool lzo/zbud [ 3.181049] Key type big_key registered [ 3.193145] Key type encrypted registered [ 3.194712] ima: No TPM chip found, activating TPM-bypass! [ 3.196656] ima: Allocated hash algorithm: sha1 [ 3.198282] ima: No architecture policies found [ 3.200187] evm: Initialising EVM extended attributes: [ 3.202177] evm: security.selinux [ 3.203396] evm: security.ima [ 3.204490] evm: security.capability [ 3.205773] evm: HMAC attrs: 0x1 [ 3.207981] rtc_cmos 00:05: setting system clock to 2026-09-14 16:33:42 UTC (1789403622) [ 3.214773] debug: unmapping init [mem 0xffffffffa4003000-0xffffffffa41fffff] [ 3.217581] debug: unmapping init [mem 0xffffffffa2d82000-0xffffffffa3058fff] [ 3.222124] Write protecting the kernel read-only data: 28672k [ 3.225465] debug: unmapping init [mem 0xffffffffa1403000-0xffffffffa15fffff] [ 3.227553] debug: unmapping init [mem 0xffffffffa1d14000-0xffffffffa1dfffff] [ 3.262354] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.272245] systemd[1]: Detected virtualization kvm. [ 3.273858] systemd[1]: Detected architecture x86-64. [ 3.275176] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.302262] systemd[1]: No hostname configured. [ 3.303699] systemd[1]: Set hostname to . [ 3.305291] random: systemd: uninitialized urandom read (16 bytes read) [ 3.307063] systemd[1]: Initializing machine ID from random generator. [ 3.447647] random: systemd: uninitialized urandom read (16 bytes read) [ 3.449848] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.455698] random: systemd: uninitialized urandom read (16 bytes read) [ 3.457315] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.460263] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.035889] device-mapper: uevent: version 1.0.3 [ 4.038156] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.703133] random: fast init done [ 4.716539] virtio_net virtio0 ens2: renamed from eth0 [ 4.779365] scsi host0: ata_piix [ 4.797217] scsi host1: ata_piix [ 4.798747] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.801491] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.213417] dracut-initqueue[581]: RTNETLINK answers: File exists [ 9.712175] random: crng init done [ 9.712904] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.083381] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.180792] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.475484] SELinux: Disabled at runtime. [ 11.544654] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.553254] systemd[1]: Detected virtualization kvm. [ 11.555095] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.054883] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.058412] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.062678] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.065819] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.068457] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.076503] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.081992] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting POSIX Message Queue File System... [ 12.214824] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.500949] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.801198] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.836833] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.879222] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.892931] EDAC sbridge: Ver: 1.1.2 [ 15.779894] Key type dns_resolver registered [ 16.604912] NFS: Registering the id_resolver key type [ 16.609409] Key type id_resolver registered [ 16.611466] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Login Service. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg206-server login: [ 86.804104] libcfs: loading out-of-tree module taints kernel. [ 86.931854] Key type ._llcrypt registered [ 86.939610] Key type .llcrypt registered [ 87.086940] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 104.146023] hrtimer: interrupt took 12006116 ns [ 108.053634] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 110.032836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 110.043556] alg: No test for adler32 (adler32-zlib) [ 111.732584] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 113.024116] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 114.887216] Key type lgssc registered [ 117.591921] Lustre: Echo OBD driver; http://www.lustre.org/ [ 134.268564] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 175.879892] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 189.063958] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 189.104924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 190.403865] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 190.433456] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 190.506986] Lustre: lustre-MDT0000: new disk, initializing [ 190.679068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 190.699730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 197.213698] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.067627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 212.194540] Lustre: 6507:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 212.224178] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 212.228093] Lustre: Skipped 1 previous similar message [ 212.411979] Lustre: lustre-MDT0001: new disk, initializing [ 212.459089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 212.476047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 212.490975] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 217.355486] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 223.212134] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 234.498598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 234.742337] Lustre: lustre-OST0000: new disk, initializing [ 234.745910] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 234.752160] Lustre: 8447:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 234.808550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 236.930705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 236.935745] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 237.047614] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 241.907951] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 259.822498] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 260.092311] Lustre: lustre-OST0001: new disk, initializing [ 260.098616] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 260.108452] Lustre: 9520:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 260.227846] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 265.804988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 265.821357] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 265.908102] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 269.357251] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 282.365809] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 289.408790] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 296.445711] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing check_logdir /tmp/testlogs/ [ 302.617427] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing yml_node [ 308.115792] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 311.311201] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 313.213390] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 314.614934] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Sep 14 12:38:52 EDT 2026 [ 330.995684] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 333.899461] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 355.807943] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 355.811537] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 355.855581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 358.368425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 358.368706] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 360.840048] Lustre: server umount lustre-MDT0000 complete [ 365.660163] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789403984 with bad export cookie 11083801584214215731 [ 365.670316] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 368.615169] LustreError: 10345:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 368.617238] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 368.677607] LustreError: 10345:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 368.708843] Lustre: Skipped 2 previous similar messages [ 368.709378] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 368.709383] Lustre: Skipped 3 previous similar messages [ 370.656146] LustreError: 6517:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 370.670753] LustreError: 6517:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 372.109697] Lustre: server umount lustre-MDT0001 complete [ 377.733506] Lustre: server umount lustre-OST0000 complete [ 382.302621] Lustre: server umount lustre-OST0001 complete [ 390.020568] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 402.147971] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 442.087143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 442.498583] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 442.612904] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 442.788075] Lustre: lustre-MDT0000: new disk, initializing [ 442.935123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 442.956451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 448.886739] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 460.385822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 460.538745] Lustre: 15074:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 460.588861] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 460.592174] Lustre: Skipped 1 previous similar message [ 460.695170] Lustre: lustre-MDT0001: new disk, initializing [ 460.821908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 460.913404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 460.923770] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 465.514262] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 475.338028] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 482.676934] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 490.460924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 490.678479] Lustre: lustre-OST0000: new disk, initializing [ 490.681611] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 490.685597] Lustre: 17043:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 490.754428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 492.063588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 492.070659] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 492.185078] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 496.914075] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 505.660982] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 511.585535] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 511.773897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 516.990085] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 517.301242] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 522.726680] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 522.736333] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 522.745620] Lustre: Skipped 1 previous similar message [ 522.748737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 522.751691] Lustre: Skipped 4 previous similar messages [ 525.692586] Lustre: server umount lustre-OST0000 complete [ 532.453060] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 532.469080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 532.478218] Lustre: Skipped 1 previous similar message [ 532.484176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 536.005101] Lustre: server umount lustre-MDT0000 complete [ 538.080228] LustreError: 15086:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 538.089824] LustreError: 15086:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 539.546967] LustreError: 15067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789404158 with bad export cookie 11083801584214217264 [ 539.561135] LustreError: 15067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 539.565287] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 539.916906] Lustre: server umount lustre-MDT0001 complete [ 544.295590] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 12:42:41 (1789404161) [ 546.227770] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 547.660808] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 12:42:45 (1789404165) [ 556.750221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 557.315723] LustreError: 19452:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 557.501169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 562.659057] LustreError: 19453:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 562.990862] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 571.880026] LustreError: 19453:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 571.934850] LustreError: 19453:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 572.522383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 578.238989] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 587.689265] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 594.647114] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 602.058329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 607.969622] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 615.549515] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 621.337158] LustreError: 19447:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023256130 [ 621.349328] LustreError: 19447:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023256130 [ 627.173055] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 627.175736] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 627.199422] Lustre: Skipped 1 previous similar message [ 627.203724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 627.203733] Lustre: Skipped 1 previous similar message [ 632.239555] Lustre: server umount lustre-OST0000 complete [ 642.534714] LustreError: 19447:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 642.543932] LustreError: 19447:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 642.567929] Lustre: server umount lustre-MDT0000 complete [ 646.789403] LustreError: 19432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789404266 with bad export cookie 11083801584214218146 [ 646.812633] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 647.432851] Lustre: server umount lustre-MDT0001 complete [ 662.294292] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 666.518194] Key type lgssc unregistered [ 666.886747] LNet: 22984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 666.895746] LNetError: 22984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 666.911590] LNet: Removed LNI 192.168.202.106@tcp [ 667.909185] Key type .llcrypt unregistered [ 667.910967] Key type ._llcrypt unregistered [ 694.188700] Key type ._llcrypt registered [ 694.194401] Key type .llcrypt registered [ 694.417604] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 696.223857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 696.276856] alg: No test for adler32 (adler32-zlib) [ 697.410488] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 697.657914] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 699.399216] Key type lgssc registered [ 700.826023] Lustre: Echo OBD driver; http://www.lustre.org/ [ 709.325968] loop: module loaded [ 709.395571] loop0: detected capacity change from 0 to 204800000 [ 709.672513] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 709.681827] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 710.146465] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 716.200194] loop0: detected capacity change from 0 to 204800000 [ 716.404198] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 716.424758] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 716.436260] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 721.548988] loop0: detected capacity change from 0 to 204800000 [ 728.095725] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 728.128793] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 729.359980] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 729.437915] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 729.523398] Lustre: lustre-MDT0000: new disk, initializing [ 729.626824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 729.654504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 735.381854] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 745.645254] loop1: detected capacity change from 0 to 204800000 [ 752.241726] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 752.693237] Lustre: lustre-OST07c6: new disk, initializing [ 752.695985] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 752.698665] Lustre: Skipped 1 previous similar message [ 752.708353] Lustre: 26170:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 752.806839] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 759.831593] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 760.963837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 760.978919] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 761.019877] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 764.640655] Lustre: 25157:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 764.650631] Lustre: 25157:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 764.653825] Lustre: 25157:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 764.658952] Lustre: 25157:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 764.665604] Lustre: 25157:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 764.670718] Lustre: 25157:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 784.072894] Lustre: 25158:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 784.082611] Lustre: 25158:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 784.094637] Lustre: 25158:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 784.109414] Lustre: 25158:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 784.118915] Lustre: 25158:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 784.126011] Lustre: 25158:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 784.136083] LustreError: 25158:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023256130 [ 784.144130] LustreError: 25158:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023256130 [ 789.601718] Lustre: 25158:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 789.608530] Lustre: 25158:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 789.616960] Lustre: 25158:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 789.624885] Lustre: 25158:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 789.631807] Lustre: 25158:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 789.639767] Lustre: 25158:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 789.644909] Lustre: 25158:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 789.650751] Lustre: 25158:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 789.655013] Lustre: 25158:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 789.660653] Lustre: 25158:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 789.664574] Lustre: 25158:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 789.676086] Lustre: 25158:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 789.683740] LustreError: 25158:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023256130 [ 789.688721] LustreError: 25158:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023256130 [ 794.508163] Lustre: 26787:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 794.516880] Lustre: 26787:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 794.522427] Lustre: 26787:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 794.532413] Lustre: 26787:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 794.539063] Lustre: 26787:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 794.546731] Lustre: 26787:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 794.556077] Lustre: 26787:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 794.567685] Lustre: 26787:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 794.575220] Lustre: 26787:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 794.590145] Lustre: 26787:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 794.600937] Lustre: 26787:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 794.610524] Lustre: 26787:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 794.624781] LustreError: 26787:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023256130 [ 794.632703] LustreError: 26787:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023256130 [ 799.660181] Lustre: 25156:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 799.675821] Lustre: 25156:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 799.686152] Lustre: 25156:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 799.695680] Lustre: 25156:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 799.722286] Lustre: 25156:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 799.732024] Lustre: 25156:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 799.753108] Lustre: 25156:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 799.760160] Lustre: 25156:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 799.772129] Lustre: 25156:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 799.780413] Lustre: 25156:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 799.789212] Lustre: 25156:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 799.796205] Lustre: 25156:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 799.805434] LustreError: 25156:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023256130 [ 799.810669] LustreError: 25156:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023256130 [ 807.794344] Lustre: 25157:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 807.803026] Lustre: 25157:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 807.809219] Lustre: 25157:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 807.816027] Lustre: 25157:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 807.821676] Lustre: 25157:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 807.827470] Lustre: 25157:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 807.836326] Lustre: 25157:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 807.844029] Lustre: 25157:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 807.853688] Lustre: 25157:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 807.861456] Lustre: 25157:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 807.867273] Lustre: 25157:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 807.872018] Lustre: 25157:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 807.878344] LustreError: 25157:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023256130 [ 807.883931] LustreError: 25157:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 1 previous similar message [ 807.890194] LustreError: 25157:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023256130 [ 807.898161] LustreError: 25157:0:(mdt_open.c:646:mdt_finish_open()) Skipped 1 previous similar message [ 819.273341] LustreError: 25156:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023256130 [ 819.282574] LustreError: 25156:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 2 previous similar messages [ 819.287211] LustreError: 25156:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023256130 [ 819.292140] LustreError: 25156:0:(mdt_open.c:646:mdt_finish_open()) Skipped 2 previous similar messages [ 824.057891] Lustre: 26787:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 824.066461] Lustre: 26787:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 7 previous similar messages [ 824.079509] Lustre: 26787:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 824.086905] Lustre: 26787:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 824.102605] Lustre: 26787:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 824.121153] Lustre: 26787:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 824.128322] Lustre: 26787:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 824.135652] Lustre: 26787:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 824.148594] Lustre: 26787:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 824.159461] Lustre: 26787:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 824.170682] Lustre: 26787:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 824.183315] Lustre: 26787:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 861.161257] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 861.191990] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 866.273753] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 866.658744] Lustre: server umount lustre-OST07c6 complete [ 877.056095] Lustre: server umount lustre-MDT0000 complete [ 894.942396] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 898.271949] Key type lgssc unregistered [ 898.713840] LNet: 28729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 898.728953] LNetError: 28729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 898.753839] LNet: Removed LNI 192.168.202.106@tcp [ 899.972300] Key type .llcrypt unregistered [ 899.974441] Key type ._llcrypt unregistered [ 919.879694] Key type ._llcrypt registered [ 919.885017] Key type .llcrypt registered [ 920.088355] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 921.450914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 921.494590] alg: No test for adler32 (adler32-zlib) [ 922.611702] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 922.872682] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 924.599247] Key type lgssc registered [ 926.299854] Lustre: Echo OBD driver; http://www.lustre.org/ [ 934.843101] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 934.876513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 936.456170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 941.795093] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 946.664026] LustreError: 29997:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 946.684276] LustreError: 29997:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 951.783047] LustreError: 29998:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 952.256436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 952.707900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 956.843735] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 965.283731] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 973.335838] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 981.725066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 982.116022] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 989.090838] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 998.312679] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1006.716379] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1012.601731] LustreError: 29993:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2207613190210 [ 1012.610788] LustreError: 29993:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2207613190210 [ 1014.557651] LustreError: 29994:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2207655133185 [ 1026.720397] Lustre: server umount lustre-OST0000 complete [ 1032.678643] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1032.684431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1032.697425] Lustre: Skipped 1 previous similar message [ 1032.712929] Lustre: Skipped 1 previous similar message [ 1037.794000] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1037.800498] Lustre: Skipped 1 previous similar message [ 1040.965116] Lustre: server umount lustre-MDT0000 complete [ 1042.790439] Lustre: 29307:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789404646/real 1789404646] req@00000000fc44ba96 x1876326657068032/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789404662 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1042.829108] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1042.856447] LustreError: 30725:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1042.875257] LustreError: 30725:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1043.938352] Lustre: 29309:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789404647/real 1789404647] req@00000000ff2b560f x1876326657068288/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789404663 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1045.139433] LustreError: 33005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789404664 with bad export cookie 5842053046853808573 [ 1045.143380] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1045.150960] LustreError: 33005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1045.830633] Lustre: server umount lustre-MDT0001 complete [ 1056.193859] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 1059.079817] Key type lgssc unregistered [ 1059.534816] LNet: 33758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1059.547419] LNetError: 33758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1059.577418] LNet: Removed LNI 192.168.202.106@tcp [ 1060.301976] Key type .llcrypt unregistered [ 1060.304374] Key type ._llcrypt unregistered [ 1085.563465] Key type ._llcrypt registered [ 1085.565553] Key type .llcrypt registered [ 1085.679904] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 1087.406954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1087.604819] alg: No test for adler32 (adler32-zlib) [ 1088.825593] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 1089.239729] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 1091.039554] Key type lgssc registered [ 1092.503460] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1102.934289] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1102.945935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1104.539334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1109.998845] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1114.097512] LustreError: 35026:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1114.121618] LustreError: 35026:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1119.213502] LustreError: 35027:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1119.337953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1119.776829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1124.965068] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1132.134964] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1137.603758] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1144.062117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1144.426190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1149.561340] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1151.074488] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1160.190970] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1165.514888] LustreError: 35022:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255619 [ 1165.517634] LustreError: 35022:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255619 [ 1170.412245] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1170.422829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1170.428845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1170.433842] Lustre: Skipped 1 previous similar message [ 1170.445952] Lustre: Skipped 1 previous similar message [ 1175.521879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1175.535975] Lustre: Skipped 1 previous similar message [ 1176.497898] Lustre: server umount lustre-OST0000 complete [ 1180.645970] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1180.663537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1180.670185] Lustre: Skipped 1 previous similar message [ 1185.762680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1185.766630] Lustre: Skipped 1 previous similar message [ 1186.889860] Lustre: server umount lustre-MDT0000 complete [ 1190.888200] LustreError: 36188:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1190.923853] LustreError: 36188:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1192.164687] LustreError: 35007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789404811 with bad export cookie 11175738247227616929 [ 1192.171690] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1192.179040] LustreError: 35007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1192.882417] Lustre: server umount lustre-MDT0001 complete [ 1205.612029] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 1209.236254] Key type lgssc unregistered [ 1209.476547] LNet: 38559:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1209.484267] LNetError: 38559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1209.496230] LNet: Removed LNI 192.168.202.106@tcp [ 1210.116512] Key type .llcrypt unregistered [ 1210.118692] Key type ._llcrypt unregistered [ 1236.670480] Key type ._llcrypt registered [ 1236.674245] Key type .llcrypt registered [ 1236.822336] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 1238.372132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1238.508065] alg: No test for adler32 (adler32-zlib) [ 1239.801824] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 1240.125953] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 1241.903339] Key type lgssc registered [ 1243.483353] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1252.275391] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1252.288421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1253.704250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1258.094968] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1263.076064] LustreError: 39990:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1263.106596] LustreError: 39990:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1266.850467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1267.221455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1271.493634] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1281.587742] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1287.604911] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1295.715399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1296.233996] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1298.317334] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1304.289311] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1313.382281] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1317.263718] LustreError: 39985:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2207613190210 [ 1317.266408] LustreError: 39985:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2207613190210 [ 1320.566233] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.6@tcp (stopping) [ 1321.767903] Lustre: server umount lustre-MDT0000 complete [ 1322.994348] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1323.022097] LustreError: 40717:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1323.028921] Lustre: Skipped 1 previous similar message [ 1323.085118] LustreError: 40717:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1325.653566] LustreError: 39984:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1325.941404] LustreError: 40725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789404945 with bad export cookie 12937396819681887441 [ 1325.951402] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1325.967874] LustreError: 40725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1328.096262] LustreError: 39986:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1328.106083] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1328.118224] Lustre: Skipped 1 previous similar message [ 1328.122418] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1330.150198] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1335.264688] LustreError: 39986:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1335.266070] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1335.301589] LustreError: 39986:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1335.320682] Lustre: Skipped 1 previous similar message [ 1340.383242] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1340.657252] Lustre: server umount lustre-MDT0001 complete [ 1357.160456] Lustre: server umount lustre-OST0000 complete [ 1375.800599] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 1379.557258] Key type lgssc unregistered [ 1380.138961] LNet: 43851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1380.157562] LNetError: 43851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1380.192053] LNet: Removed LNI 192.168.202.106@tcp [ 1381.396395] Key type .llcrypt unregistered [ 1381.400035] Key type ._llcrypt unregistered [ 1406.654170] Key type ._llcrypt registered [ 1406.660629] Key type .llcrypt registered [ 1406.838273] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 1408.216637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1408.285555] alg: No test for adler32 (adler32-zlib) [ 1409.341839] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 1409.609887] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 1411.327234] Key type lgssc registered [ 1412.693986] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1421.619750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1421.634838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1423.191043] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1429.569408] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1433.575248] LustreError: 45265:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1433.617219] LustreError: 45265:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1438.604271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1438.688342] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1438.822828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1444.051500] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1452.469838] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1458.197688] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1466.244035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1466.756801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1467.766313] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1472.824847] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1480.887584] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1488.114049] LustreError: 45260:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2207613190210 [ 1488.137220] LustreError: 45260:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2207613190210 [ 1493.108478] Lustre: lustre-OST0000: Not available for connect from 192.168.202.6@tcp (stopping) [ 1493.471936] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1493.494665] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1494.507777] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1494.515053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1494.515840] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1494.539023] Lustre: Skipped 2 previous similar messages [ 1496.531210] Lustre: server umount lustre-OST0000 complete [ 1519.079396] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1519.093592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1519.094289] Lustre: Skipped 1 previous similar message [ 1519.109376] Lustre: Skipped 1 previous similar message [ 1522.031087] Lustre: server umount lustre-MDT0000 complete [ 1524.200961] LustreError: 45264:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1524.219064] LustreError: 45264:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1526.095102] LustreError: 45246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789405145 with bad export cookie 3672742733088271061 [ 1526.099403] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1526.122682] LustreError: 45246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1532.747053] Lustre: server umount lustre-MDT0001 complete [ 1545.277602] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 1548.510424] Key type lgssc unregistered [ 1548.951777] LNet: 48975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1548.961210] LNetError: 48975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1550.006760] LNet: Removed LNI 192.168.202.106@tcp [ 1550.998543] Key type .llcrypt unregistered [ 1551.006053] Key type ._llcrypt unregistered [ 1574.974896] Key type ._llcrypt registered [ 1574.979202] Key type .llcrypt registered [ 1575.116339] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 1576.491056] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1576.593096] alg: No test for adler32 (adler32-zlib) [ 1577.729676] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 1577.968252] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 1579.655215] Key type lgssc registered [ 1581.087954] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1589.830826] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1589.857278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1591.348314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1596.709268] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1601.509174] LustreError: 50388:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1601.516886] LustreError: 50388:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1604.610540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1604.853697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1609.242677] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1617.243499] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1625.054721] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1632.094203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1632.415967] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1636.529328] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1638.606748] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1646.121871] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1655.113039] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1658.644172] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1671.400300] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-1413h01m28s [ 1673.088885] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1675.758391] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1675.779853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1675.788681] Lustre: Skipped 1 previous similar message [ 1676.842824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1676.847774] Lustre: Skipped 4 previous similar messages [ 1678.444494] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.6@tcp (stopping) [ 1681.699664] Lustre: server umount lustre-MDT0000 complete [ 1682.914740] LustreError: 50384:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1682.941798] LustreError: 50384:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1685.677166] LustreError: 50384:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1686.661581] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1689.066612] LustreError: 50407:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1689.095852] LustreError: 50407:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1695.209806] LustreError: 50388:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1695.223700] LustreError: 50388:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1695.834135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1696.089601] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1696.632414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1696.717302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1697.648896] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1703.316884] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1707.476971] Lustre: DEBUG MARKER: Wait for df (36672) ... [ 1709.288663] Lustre: DEBUG MARKER: done [ 1716.705041] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1716.723293] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1716.755043] Lustre: Skipped 1 previous similar message [ 1716.764442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1716.766907] Lustre: Skipped 1 previous similar message [ 1717.216492] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1720.511357] Lustre: server umount lustre-OST0000 complete [ 1728.481535] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1728.496643] Lustre: Skipped 1 previous similar message [ 1728.504447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1728.514084] Lustre: Skipped 4 previous similar messages [ 1731.679096] Lustre: server umount lustre-MDT0000 complete [ 1735.651217] LustreError: 50388:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1735.665058] LustreError: 50388:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1736.375942] LustreError: 50369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789405355 with bad export cookie 9769616139732219219 [ 1736.381748] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1736.384735] LustreError: 50369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1737.015022] Lustre: server umount lustre-MDT0001 complete [ 1751.582720] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 1755.176603] Key type lgssc unregistered [ 1755.563658] LNet: 55328:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1755.572240] LNetError: 55328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1755.592582] LNet: Removed LNI 192.168.202.106@tcp [ 1756.843400] Key type .llcrypt unregistered [ 1756.848278] Key type ._llcrypt unregistered [ 1800.985884] Key type ._llcrypt registered [ 1800.988089] Key type .llcrypt registered [ 1801.119532] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 1802.745188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1802.812401] alg: No test for adler32 (adler32-zlib) [ 1803.957764] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 1804.259112] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 1806.065345] Key type lgssc registered [ 1807.492804] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1815.558308] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1815.608028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1816.957469] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1816.976765] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1817.138842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1822.340835] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1831.269985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1831.359253] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1831.431348] Lustre: 57532:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1831.661525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1836.058214] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1843.891477] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1850.684170] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1858.719694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1859.060487] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1859.293852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1865.947437] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1866.906642] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1875.142971] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1882.725143] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-1413h05m00s [ 1884.711861] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1892.893380] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1895.890944] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1895.936902] LustreError: 59405:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1916.047482] Lustre: lustre-MDT0000: Client c80c0ef7-3fbe-46cf-8c9d-9793289e2883 (at 192.168.202.6@tcp) reconnecting [ 1936.015394] LustreError: 59405:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1936.068468] LustreError: 57541:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2207613190210 [ 1936.077168] LustreError: 57541:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2207613190210 [ 1938.443151] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1940.980448] Lustre: DEBUG MARKER: done [ 1947.656590] Lustre: server umount lustre-OST0000 complete [ 1953.250553] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1953.253727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1953.273348] Lustre: Skipped 1 previous similar message [ 1954.338473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1954.346124] Lustre: Skipped 2 previous similar messages [ 1960.422626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1962.544329] Lustre: server umount lustre-MDT0000 complete [ 1964.512219] Lustre: 56857:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789405567/real 1789405567] req@00000000bbe9dd92 x1876327580884096/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789405583 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1964.564925] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1967.091677] LustreError: 57526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789405586 with bad export cookie 10334905623295787450 [ 1967.100064] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1967.100719] LustreError: 57526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1967.955746] Lustre: server umount lustre-MDT0001 complete [ 1978.852555] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 1981.232619] Key type lgssc unregistered [ 1981.761870] LNet: 61873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1981.774074] LNetError: 61873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1981.796163] LNet: Removed LNI 192.168.202.106@tcp [ 1982.838961] Key type .llcrypt unregistered [ 1982.845141] Key type ._llcrypt unregistered [ 2035.364959] Key type ._llcrypt registered [ 2035.367393] Key type .llcrypt registered [ 2035.565318] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 2037.532465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2037.569741] alg: No test for adler32 (adler32-zlib) [ 2038.755602] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 2039.175259] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 2040.927321] Key type lgssc registered [ 2042.396748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2046.600333] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2046.657796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2048.200797] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2048.257740] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2048.597973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2049.762642] Lustre: Failing over lustre-MDT0000 [ 2050.187274] Lustre: server umount lustre-MDT0000 complete [ 2057.654538] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 13:07:55 (1789405675) [ 2065.128751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2065.501871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2070.623848] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2080.426093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2080.583072] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2080.607701] Lustre: 64669:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2080.879585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2085.385410] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2092.571385] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2099.850722] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2107.670918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2107.928120] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2108.153238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2116.042798] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2116.332537] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2125.317454] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2131.784550] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2132.022069] LustreError: 64675:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2207613255681 [ 2132.082303] LustreError: 64675:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2207613190338 [ 2132.532483] LustreError: 64675:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2207613190338 [ 2132.537778] LustreError: 64675:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 3 previous similar messages [ 2132.622864] LustreError: 64677:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2207613190338 [ 2132.627342] LustreError: 64677:0:(mdt_open.c:646:mdt_finish_open()) Skipped 3 previous similar messages [ 2142.195195] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2142.201215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2142.202062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2142.265316] Lustre: Skipped 1 previous similar message [ 2145.690880] Lustre: server umount lustre-MDT0000 complete [ 2149.976684] LustreError: 64662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789405769 with bad export cookie 17009887613920815537 [ 2149.979041] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2149.991120] LustreError: 64662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2150.336768] Lustre: server umount lustre-MDT0001 complete [ 2153.915909] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2159.420702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2167.712184] Lustre: 63589:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789405771/real 1789405771] req@0000000019b7f057 x1876327827285504/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789405787 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2167.752481] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2167.764896] Lustre: Skipped 1 previous similar message [ 2169.182161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2169.823334] Lustre: 63591:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789405773/real 1789405773] req@000000006c088e2f x1876327827285632/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789405789 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2172.895771] Lustre: 63592:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789405776/real 1789405776] req@00000000e95af13c x1876327827285760/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789405792 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2174.943973] LustreError: 63588:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000bac5bcaa x1876327827286528/t0(0) o250->MGC192.168.202.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2175.351421] LustreError: 66519:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2175.375834] LustreError: 66519:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2175.475519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2175.630958] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2176.418943] Lustre: 63589:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789405779/real 1789405779] req@000000005c2dc8d5 x1876327827286016/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789405795 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2176.484351] LustreError: 66520:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2178.531184] LustreError: 66519:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2178.553291] LustreError: 66519:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2180.028603] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2182.559330] Lustre: 63590:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789405785/real 1789405785] req@000000009a5b5d6a x1876327827286144/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789405801 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2183.654257] LustreError: 68442:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2183.673623] LustreError: 68442:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2188.611189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2188.782588] LustreError: 68440:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2188.806493] LustreError: 68440:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2189.237522] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2194.421444] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2194.762572] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2197.550878] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2203.434619] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2209.721774] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2210.667613] LustreError: 68441:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199065198593 [ 2210.679181] LustreError: 68441:0:(mdt_open.c:646:mdt_finish_open()) Skipped 5 previous similar messages [ 2212.158051] LustreError: 68440:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2207613190210 [ 2212.169460] LustreError: 68440:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 6 previous similar messages [ 2217.095467] Lustre: server umount lustre-MDT0000 complete [ 2217.954651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2217.957392] LustreError: 68441:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2217.970347] Lustre: Skipped 2 previous similar messages [ 2218.001585] LustreError: 68441:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2221.555584] LustreError: 64662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789405840 with bad export cookie 17009887613920817812 [ 2221.561519] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2221.570528] LustreError: 64662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2221.986573] Lustre: server umount lustre-MDT0001 complete [ 2229.140874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2239.200196] Lustre: 63592:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789405842/real 1789405842] req@00000000544088da x1876327827313664/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789405858 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2239.215535] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2239.817045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2246.431585] LustreError: 63588:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004f14c4dd x1876327827314816/t0(0) o250->MGC192.168.202.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2246.846242] LustreError: 69176:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2246.930265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2247.011958] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2251.726075] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2255.327378] Lustre: 63589:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789405857/real 1789405857] req@000000003395dd0f x1876327827314560/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789405873 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2255.349703] Lustre: 63589:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 2258.430880] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2261.113701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2266.496758] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2266.597657] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2274.156590] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2280.107198] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2281.130062] LustreError: 71122:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199065198593 [ 2281.144988] LustreError: 71122:0:(mdt_open.c:646:mdt_finish_open()) Skipped 10 previous similar messages [ 2282.057109] LustreError: 71122:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2207613190210 [ 2292.194303] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2292.196361] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2292.201941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2292.220538] Lustre: Skipped 1 previous similar message [ 2292.251892] Lustre: Skipped 4 previous similar messages [ 2293.777393] Lustre: server umount lustre-MDT0000 complete [ 2297.315082] LustreError: 72115:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2297.347341] LustreError: 72115:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 2297.997483] LustreError: 64660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789405917 with bad export cookie 17009887613920819191 [ 2298.010880] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2298.013582] LustreError: 64660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2298.293440] Lustre: server umount lustre-MDT0001 complete [ 2305.588609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2309.422270] Lustre: DEBUG MARKER: files compared the same [ 2321.806526] Lustre: server umount lustre-OST0000 complete [ 2341.420608] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 2344.930326] Key type lgssc unregistered [ 2345.327050] LNet: 74398:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2345.338746] LNetError: 74398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2345.373724] LNet: Removed LNI 192.168.202.106@tcp [ 2346.633694] Key type .llcrypt unregistered [ 2346.636708] Key type ._llcrypt unregistered [ 2373.724550] Key type ._llcrypt registered [ 2373.728256] Key type .llcrypt registered [ 2373.861776] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 2375.181735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2375.295558] alg: No test for adler32 (adler32-zlib) [ 2376.421733] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 2376.730604] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 2378.431217] Key type lgssc registered [ 2379.775388] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2388.202512] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2388.259579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2389.775539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2394.708086] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug +malloc all [ 2400.231139] LustreError: 75812:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2400.244269] LustreError: 75812:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2404.855872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2405.263222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2410.453763] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug +malloc all [ 2417.077743] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2422.976483] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2431.869597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2432.222189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2435.321674] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2439.968555] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug +malloc all [ 2449.160986] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2466.283498] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2466.290258] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2466.293240] LustreError: Skipped 1 previous similar message [ 2466.298061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2466.343121] Lustre: Skipped 1 previous similar message [ 2468.931886] Lustre: server umount lustre-OST0000 complete [ 2476.512096] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2476.538587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2476.544233] Lustre: Skipped 2 previous similar messages [ 2479.304797] Lustre: server umount lustre-MDT0000 complete [ 2481.632860] LustreError: 76540:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2481.650605] LustreError: 76540:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2483.552179] LustreError: 75794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789406102 with bad export cookie 838568611365891151 [ 2483.571874] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2484.104078] Lustre: server umount lustre-MDT0001 complete [ 2495.151894] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 2498.325609] Key type lgssc unregistered [ 2498.650594] LNet: 79366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2498.662429] LNetError: 79366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2498.675882] LNet: Removed LNI 192.168.202.106@tcp [ 2499.645572] Key type .llcrypt unregistered [ 2499.648628] Key type ._llcrypt unregistered [ 2525.876462] Key type ._llcrypt registered [ 2525.878899] Key type .llcrypt registered [ 2526.024445] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 2527.170705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2527.209440] alg: No test for adler32 (adler32-zlib) [ 2528.420210] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 2528.699178] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 2530.368242] Key type lgssc registered [ 2531.583657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2539.911770] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2539.935708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2555.423682] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2560.544492] LustreError: 80757:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.106@tcp: failed processing log, type 4: rc = -110 [ 2587.167246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2593.771193] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2601.325558] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2609.765252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2610.019505] Lustre: *** cfs_fail_loc=706, val=0*** [ 2610.524853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2610.607852] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2615.558396] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2615.790905] LustreError: 80783:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2615.812259] LustreError: 80783:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2620.912601] LustreError: 80782:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2621.927868] LustreError: 81843:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2621.949075] LustreError: 81843:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2624.371522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2624.613834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2629.096961] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2636.881581] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2642.611465] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2644.963027] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2644.979556] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2644.991974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2645.484440] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2645.507949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2645.520154] Lustre: Skipped 1 previous similar message [ 2650.594088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2650.783494] Lustre: server umount lustre-OST0000 complete [ 2655.201140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2655.214612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2655.230373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2655.232854] Lustre: Skipped 1 previous similar message [ 2660.837081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2660.847486] Lustre: Skipped 2 previous similar messages [ 2661.125708] Lustre: server umount lustre-MDT0000 complete [ 2665.425066] LustreError: 80764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789406284 with bad export cookie 16080429176113839594 [ 2665.428332] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2665.431242] LustreError: 80764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2666.057535] Lustre: server umount lustre-MDT0001 complete [ 2677.648782] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 2680.410608] Key type lgssc unregistered [ 2680.702549] LNet: 84370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2680.706390] LNetError: 84370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2680.720527] LNet: Removed LNI 192.168.202.106@tcp [ 2681.487370] Key type .llcrypt unregistered [ 2681.488618] Key type ._llcrypt unregistered [ 2711.458377] Key type ._llcrypt registered [ 2711.461812] Key type .llcrypt registered [ 2711.584893] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 2713.390249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2713.433087] alg: No test for adler32 (adler32-zlib) [ 2714.523133] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 2714.830748] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 2716.503842] Key type lgssc registered [ 2717.741714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2726.263731] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2726.284778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2731.206794] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2738.626535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2739.086538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2744.302632] LustreError: 86472:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2744.325106] LustreError: 86472:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2744.482911] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2749.410317] LustreError: 86475:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2749.447782] LustreError: 86475:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2754.533462] LustreError: 86476:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2754.550257] LustreError: 86476:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2755.645353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2756.112904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2761.187217] LustreError: 86470:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2761.212053] LustreError: 86470:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 2764.045066] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2769.378041] LustreError: 86475:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2769.394311] LustreError: 86475:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 2772.404053] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2772.420126] Lustre: MGS: Client 441b1c9e-42af-4534-951a-701c76b84803 (at 0@lo) reconnecting [ 2772.446559] LustreError: 86486:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2772.740363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2772.809755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2777.580994] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2787.214525] LustreError: 86471:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023256130 [ 2787.223669] LustreError: 86471:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023256130 [ 2793.440040] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2793.442230] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2793.472770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2793.480556] Lustre: Skipped 1 previous similar message [ 2798.561877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2798.576508] Lustre: Skipped 2 previous similar messages [ 2798.973379] Lustre: server umount lustre-OST0000 complete [ 2803.679940] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2803.682582] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2803.686882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2803.697320] Lustre: Skipped 1 previous similar message [ 2803.705793] Lustre: Skipped 1 previous similar message [ 2808.736079] Lustre: server umount lustre-MDT0000 complete [ 2808.802791] LustreError: 88017:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2808.817931] LustreError: 88017:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2812.930776] Lustre: server umount lustre-MDT0001 complete [ 2817.241694] Lustre: server umount MGS complete [ 2831.185416] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 13:20:49 (1789406449) [ 2863.626573] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 2874.710045] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 2914.128538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2920.001320] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2930.198125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2930.571856] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2930.654991] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2930.811980] Lustre: lustre-MDT0001: new disk, initializing [ 2930.894380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2930.924898] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2930.939382] Lustre: 93665:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2930.951400] LustreError: 93665:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2936.131794] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2947.215824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2947.578614] Lustre: lustre-OST0000: new disk, initializing [ 2947.583305] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2947.599269] Lustre: 94431:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2947.692913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2954.560882] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2958.437150] Lustre: 94887:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000004c8c0864 x1876328536121472/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789406593 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2965.258868] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2972.364134] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2972.394734] Lustre: MGS: Client d7537a50-b296-4fdc-a675-4fd70e34cc26 (at 0@lo) reconnecting [ 2972.459790] Lustre: 92933:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2972.488163] LustreError: 93651:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2972.528542] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2972.689845] Lustre: lustre-MDT0000: new disk, initializing [ 2972.892297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2972.920197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2972.945387] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2973.987353] Lustre: 95499:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000004f14c4dd x1876328536128000/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1789406609 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2977.972386] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2979.231565] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2983.156793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2983.172397] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2983.314668] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2993.476751] LustreError: 95925:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023256130 [ 2993.483889] LustreError: 95925:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023256130 [ 2993.494937] LustreError: 95925:0:(mdt_open.c:646:mdt_finish_open()) Skipped 1 previous similar message [ 3011.553492] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3011.566166] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3011.582983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3017.706452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3017.722026] Lustre: Skipped 3 previous similar messages [ 3024.351127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3024.643440] Lustre: server umount lustre-OST0000 complete [ 3033.066836] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3033.077589] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3033.080839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3033.080848] Lustre: Skipped 2 previous similar messages [ 3033.089803] Lustre: Skipped 2 previous similar messages [ 3034.964071] Lustre: server umount lustre-MDT0000 complete [ 3038.199174] LustreError: 95925:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3038.224780] LustreError: 95925:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3045.213526] Lustre: server umount lustre-MDT0001 complete [ 3048.652600] Lustre: server umount MGS complete [ 3060.936308] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 13:24:38 (1789406678) [ 3078.506762] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 3081.907924] Key type lgssc unregistered [ 3082.287904] LNet: 97999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3082.300425] LNetError: 97999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3082.314690] LNet: Removed LNI 192.168.202.106@tcp [ 3083.268134] Key type .llcrypt unregistered [ 3083.277034] Key type ._llcrypt unregistered [ 3103.340130] Key type ._llcrypt registered [ 3103.342124] Key type .llcrypt registered [ 3103.447480] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 3104.538849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3104.558404] alg: No test for adler32 (adler32-zlib) [ 3106.009462] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 3106.501370] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 3108.272691] Key type lgssc registered [ 3109.634931] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3118.552886] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3118.604187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3120.036863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3124.603419] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3130.353920] LustreError: 99221:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3130.383226] LustreError: 99221:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3134.594312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3135.158841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3141.659283] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3150.701966] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3157.876671] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3165.245684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3165.728446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3172.376714] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3180.559973] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3190.331208] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3200.483035] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3200.490345] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3200.505563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3200.514555] Lustre: Skipped 1 previous similar message [ 3200.538891] Lustre: Skipped 1 previous similar message [ 3205.414992] Lustre: server umount lustre-OST0000 complete [ 3210.724590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3210.727306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3210.739282] Lustre: Skipped 1 previous similar message [ 3215.844319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3215.850129] Lustre: Skipped 2 previous similar messages [ 3216.522711] Lustre: server umount lustre-MDT0000 complete [ 3219.957556] LustreError: 99221:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3219.994504] LustreError: 99221:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3220.590580] LustreError: 99201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789406839 with bad export cookie 14235101180827592027 [ 3220.599298] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3221.229495] Lustre: server umount lustre-MDT0001 complete [ 3233.547621] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 3236.757810] Key type lgssc unregistered [ 3237.043655] LNet: 102777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3237.054685] LNetError: 102777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3237.089124] LNet: Removed LNI 192.168.202.106@tcp [ 3238.575529] Key type .llcrypt unregistered [ 3238.578373] Key type ._llcrypt unregistered [ 3258.086444] Key type ._llcrypt registered [ 3258.091595] Key type .llcrypt registered [ 3258.245307] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 3259.811202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3260.039311] alg: No test for adler32 (adler32-zlib) [ 3261.123124] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 3261.392490] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 3263.103285] Key type lgssc registered [ 3264.704455] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3271.858637] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3271.870859] LustreError: 103854:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3271.888233] LustreError: 103857:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3271.904674] LustreError: 103854:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4971 [ 3271.924548] LustreError: 103854:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3271.945770] LustreError: 103854:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3271.986249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3273.501414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3281.861599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3282.251697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3287.351514] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3293.013308] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing lsmod [ 3296.469402] LustreError: 105057:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3296.516653] LustreError: 105063:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3296.522573] LustreError: 105057:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4967 [ 3296.526377] LustreError: 105057:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3296.539349] LustreError: 105057:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3296.549470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3297.067885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3298.112694] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3302.881314] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3302.885941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3302.893413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3303.400823] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3303.402758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3303.407590] Lustre: Skipped 1 previous similar message [ 3303.418973] Lustre: Skipped 2 previous similar messages [ 3306.255541] Lustre: server umount lustre-MDT0000 complete [ 3307.498081] LustreError: 103894:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3307.523187] LustreError: 103894:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3310.008503] LustreError: 103874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789406929 with bad export cookie 14426952500442708679 [ 3310.015313] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3310.027160] LustreError: 103874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3310.376782] Lustre: server umount lustre-MDT0001 complete [ 3323.061905] Lustre: server umount lustre-OST0000 complete [ 3332.194270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3332.905942] LustreError: 106127:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3333.021822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3338.131252] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3338.212957] LustreError: 106128:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3343.344654] LustreError: 106127:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3347.654445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3347.917102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3352.996241] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3361.165938] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3367.930649] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3376.496981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3377.037466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3378.059439] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3385.423715] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3394.696116] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3396.798551] LustreError: 106124:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255746 [ 3396.811772] LustreError: 106124:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255746 [ 3403.767355] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3403.772213] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3403.777735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3403.781433] Lustre: Skipped 1 previous similar message [ 3403.811093] Lustre: Skipped 1 previous similar message [ 3408.248031] Lustre: server umount lustre-OST0000 complete [ 3413.984233] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3413.989782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3414.007242] Lustre: Skipped 1 previous similar message [ 3414.023048] Lustre: Skipped 1 previous similar message [ 3419.106138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3419.115740] Lustre: Skipped 1 previous similar message [ 3419.520513] Lustre: server umount lustre-MDT0000 complete [ 3423.660172] LustreError: 106107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789407042 with bad export cookie 14426952500442709610 [ 3423.675158] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3424.082942] Lustre: server umount lustre-MDT0001 complete [ 3436.532688] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 3440.130864] Key type lgssc unregistered [ 3440.642288] LNet: 109661:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3440.648566] LNetError: 109661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3440.680281] LNet: Removed LNI 192.168.202.106@tcp [ 3442.000185] Key type .llcrypt unregistered [ 3442.003197] Key type ._llcrypt unregistered [ 3467.616758] Key type ._llcrypt registered [ 3467.619896] Key type .llcrypt registered [ 3467.729778] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 3469.061211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3469.244489] alg: No test for adler32 (adler32-zlib) [ 3470.331750] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 3470.526117] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 3472.159274] Key type lgssc registered [ 3473.181356] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3480.656459] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3480.673882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3482.300512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3488.472280] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3491.882733] LustreError: 111075:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3491.931174] LustreError: 111075:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3496.933301] LustreError: 111076:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3498.487568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3498.869809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3503.534768] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3508.985696] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3516.283992] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3523.677629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3523.974087] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3527.113490] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3531.080985] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3539.457520] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3544.584799] LustreError: 111070:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255746 [ 3544.597109] LustreError: 111070:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255746 [ 3545.941888] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3550.513412] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3555.809344] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3555.815182] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3555.823972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3557.869468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3557.870109] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3557.871188] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3562.098073] Lustre: server umount lustre-OST0000 complete [ 3568.097320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3568.104370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3568.119390] Lustre: Skipped 1 previous similar message [ 3568.129203] Lustre: Skipped 2 previous similar messages [ 3573.216724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3573.227364] Lustre: Skipped 1 previous similar message [ 3578.337909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3580.387266] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3580.594929] Lustre: server umount lustre-MDT0000 complete [ 3583.477866] LustreError: 113568:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3583.503648] LustreError: 113568:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3584.843717] LustreError: 111056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789407204 with bad export cookie 10336459928029102649 [ 3584.854795] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3585.222373] Lustre: server umount lustre-MDT0001 complete [ 3597.449359] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 3600.258262] Key type lgssc unregistered [ 3600.545957] LNet: 114716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3600.558849] LNetError: 114716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3600.584070] LNet: Removed LNI 192.168.202.106@tcp [ 3601.603614] Key type .llcrypt unregistered [ 3601.605703] Key type ._llcrypt unregistered [ 3619.431519] Key type ._llcrypt registered [ 3619.433700] Key type .llcrypt registered [ 3619.614574] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 3621.218720] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3621.256572] alg: No test for adler32 (adler32-zlib) [ 3622.669682] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 3622.918176] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 3624.600604] Key type lgssc registered [ 3626.628765] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3638.182666] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3638.221931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3639.752725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3644.547206] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3650.026579] LustreError: 115932:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3650.048670] LustreError: 115932:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3653.718254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3654.122603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3658.303912] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3669.688614] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3676.160885] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3683.790482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3684.277456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3685.312055] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3690.848542] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3698.499407] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3702.409474] LustreError: 115927:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255746 [ 3702.418849] LustreError: 115927:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255746 [ 3709.925677] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3709.932284] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3709.951172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3712.442414] Lustre: server umount lustre-OST0000 complete [ 3720.161785] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3720.184031] Lustre: Skipped 1 previous similar message [ 3720.193437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3720.200182] Lustre: Skipped 1 previous similar message [ 3723.749633] Lustre: server umount lustre-MDT0000 complete [ 3725.283295] LustreError: 115932:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3725.302615] LustreError: 115932:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3728.374202] LustreError: 115914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789407347 with bad export cookie 13495856212713922482 [ 3728.380593] LustreError: 115914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3728.390907] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3728.724766] Lustre: server umount lustre-MDT0001 complete [ 3741.680178] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 3744.754628] Key type lgssc unregistered [ 3745.130385] LNet: 119468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3745.135985] LNetError: 119468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3745.156450] LNet: Removed LNI 192.168.202.106@tcp [ 3746.326397] Key type .llcrypt unregistered [ 3746.329346] Key type ._llcrypt unregistered [ 3772.979685] Key type ._llcrypt registered [ 3772.981257] Key type .llcrypt registered [ 3773.048987] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 3774.054127] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3774.128948] alg: No test for adler32 (adler32-zlib) [ 3775.219679] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 3775.600354] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 3777.343260] Key type lgssc registered [ 3778.586297] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3788.748217] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3788.763538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3790.270461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3795.646838] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3800.544885] LustreError: 120885:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3800.568643] LustreError: 120885:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3801.571528] LustreError: 120884:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3804.675850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3805.128420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3810.185382] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3817.048680] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3823.011119] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3829.473669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3829.904141] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3835.358984] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3836.020258] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3841.680891] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3848.999784] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3853.185436] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3853.189634] Lustre: Skipped 2 previous similar messages [ 3854.559148] LustreError: 120879:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023256130 [ 3854.564062] LustreError: 120879:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023256130 [ 3854.934624] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3854.988426] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3855.121901] LustreError: 120879:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2207613190210 [ 3855.126260] LustreError: 120879:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 1 previous similar message [ 3855.130889] LustreError: 120879:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2207613190210 [ 3855.134203] LustreError: 120879:0:(mdt_open.c:646:mdt_finish_open()) Skipped 1 previous similar message [ 3858.556629] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3858.561760] Lustre: Skipped 2 previous similar messages [ 3861.206426] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3861.217035] Lustre: Skipped 1 previous similar message [ 3862.687911] LustreError: 123634:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199065198593 [ 3862.697032] LustreError: 123634:0:(mdt_open.c:646:mdt_finish_open()) Skipped 1 previous similar message [ 3863.097418] LustreError: 123634:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255553 [ 3863.106944] LustreError: 123634:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 1 previous similar message [ 3867.588446] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.6@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3867.595618] Lustre: Skipped 2 previous similar messages [ 3875.447789] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.6@tcp 192.168.0.[2,10]@tcp [ 3876.885925] LustreError: 120881:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199065198593 [ 3876.889059] LustreError: 120881:0:(mdt_open.c:646:mdt_finish_open()) Skipped 2 previous similar messages [ 3877.041866] LustreError: 120881:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2207613190210 [ 3879.692310] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.6/24@tcp in log lustre-MDT0000 [ 3879.702386] Lustre: Skipped 2 previous similar messages [ 3881.429243] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.6/24@tcp [ 3881.432255] Lustre: Skipped 1 previous similar message [ 3883.101772] LustreError: 120880:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199065198593 [ 3883.108046] LustreError: 120880:0:(mdt_open.c:646:mdt_finish_open()) Skipped 3 previous similar messages [ 3883.331970] LustreError: 120880:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2207613190210 [ 3883.336661] LustreError: 120880:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 1 previous similar message [ 3889.129241] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3889.134381] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3889.141763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3892.710818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3892.711841] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3894.040944] Lustre: server umount lustre-OST0000 complete [ 3897.828905] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3897.848174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3897.853074] Lustre: Skipped 1 previous similar message [ 3902.435567] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3902.439414] LustreError: Skipped 3 previous similar messages [ 3902.443152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3902.459794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3908.064632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3908.070986] Lustre: Skipped 2 previous similar messages [ 3912.159137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3912.629403] Lustre: server umount lustre-MDT0000 complete [ 3913.188462] LustreError: 120880:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3913.212808] LustreError: 120880:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3916.712616] LustreError: 120866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789407536 with bad export cookie 7765106946220508358 [ 3916.714776] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3916.720220] LustreError: 120866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3916.930339] Lustre: server umount lustre-MDT0001 complete [ 3927.766949] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 3931.049325] Key type lgssc unregistered [ 3931.366582] LNet: 125734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3931.375188] LNetError: 125734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3931.414771] LNet: Removed LNI 192.168.202.106@tcp [ 3932.438051] Key type .llcrypt unregistered [ 3932.440391] Key type ._llcrypt unregistered [ 3964.091181] Key type ._llcrypt registered [ 3964.093822] Key type .llcrypt registered [ 3964.194960] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 3965.777029] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3965.804425] alg: No test for adler32 (adler32-zlib) [ 3966.928397] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 3967.247252] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 3968.960352] Key type lgssc registered [ 3970.162212] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3978.441150] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3978.478907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3980.007669] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3980.039748] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.6/24@tcp [ 3980.055274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3985.066559] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3990.496846] LustreError: 127296:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3990.511871] LustreError: 127296:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3993.879601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3994.310497] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3994.354525] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.6/24@tcp [ 3994.393187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3999.697857] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4008.018316] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4013.912574] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4021.087440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4021.547117] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4025.679592] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 4029.055932] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4036.699906] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4041.447613] LustreError: 127293:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255746 [ 4041.453061] LustreError: 127293:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255746 [ 4046.306550] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4046.322081] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4046.351807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4049.377686] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4049.383526] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4049.387273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4049.404421] Lustre: Skipped 1 previous similar message [ 4052.463719] Lustre: server umount lustre-OST0000 complete [ 4059.616477] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4059.643906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4063.590975] Lustre: server umount lustre-MDT0000 complete [ 4064.745050] LustreError: 127297:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4064.761776] LustreError: 127297:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4067.536057] LustreError: 127276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789407686 with bad export cookie 6442247002537204054 [ 4067.544526] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4067.555982] LustreError: 127276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4068.085859] Lustre: server umount lustre-MDT0001 complete [ 4080.986491] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 4083.750701] Key type lgssc unregistered [ 4084.125347] LNet: 130884:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4084.133513] LNetError: 130884:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4084.160317] LNet: Removed LNI 192.168.202.106@tcp [ 4085.298148] Key type .llcrypt unregistered [ 4085.304696] Key type ._llcrypt unregistered [ 4110.852954] Key type ._llcrypt registered [ 4110.855949] Key type .llcrypt registered [ 4111.004707] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 4127.813260] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 4128.614535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4128.673587] alg: No test for adler32 (adler32-zlib) [ 4129.768600] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 4129.910675] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 4131.559903] Key type lgssc registered [ 4133.178318] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4175.461706] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4175.487666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4176.677103] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4176.702605] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4176.800170] Lustre: lustre-MDT0000: new disk, initializing [ 4176.884777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4176.894670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4182.514641] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4194.619194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4194.782856] Lustre: 134284:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4194.885959] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4194.897226] Lustre: Skipped 1 previous similar message [ 4195.011832] Lustre: lustre-MDT0001: new disk, initializing [ 4195.171807] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4195.220493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4195.238378] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4200.329954] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4211.563085] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4217.725410] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4225.222863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4225.536408] Lustre: lustre-OST0000: new disk, initializing [ 4225.540616] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4225.550538] Lustre: 136252:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4225.673381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4231.937452] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4232.289803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4232.298611] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4232.413032] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4242.428141] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4247.176297] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4247.419365] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4253.335266] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4253.696834] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4258.279431] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4258.298904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4258.300618] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4258.307775] Lustre: Skipped 1 previous similar message [ 4258.349210] Lustre: Skipped 1 previous similar message [ 4262.335720] Lustre: server umount lustre-OST0000 complete [ 4266.980383] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4266.985660] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4266.994914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4268.517236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4268.522464] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4272.785045] Lustre: server umount lustre-MDT0000 complete [ 4276.126345] LustreError: 134279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789407895 with bad export cookie 7210815415460706972 [ 4276.139495] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4276.143337] LustreError: 134279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4276.660940] Lustre: server umount lustre-MDT0001 complete [ 4287.299847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4287.984548] LustreError: 138471:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4288.009827] LustreError: 138471:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4288.132669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4293.601479] LustreError: 138472:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4293.614204] LustreError: 138472:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4294.021341] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4298.724956] LustreError: 138471:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4302.887492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4303.263887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4307.567436] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4315.427930] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4321.548221] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4328.756241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4329.202163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4337.394641] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4345.814395] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4351.420938] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4351.745599] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4356.552945] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4356.810324] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4367.186489] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4367.484669] Lustre: lustre-OST0001: new disk, initializing [ 4367.488346] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4367.495490] Lustre: 141847:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4367.605222] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4374.555082] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4376.697835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4376.719686] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4376.818320] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4383.392484] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4383.609544] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4389.315527] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4389.593760] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4396.170332] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8c82859f8000.ost_server_uuid 50 [ 4397.984738] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8c82859f8000.ost_server_uuid in IDLE state after 0 sec [ 4398.865863] LustreError: 138468:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023256130 [ 4398.871126] LustreError: 138468:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023256130 [ 4407.782176] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4407.792494] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4407.799562] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4407.799569] Lustre: Skipped 1 previous similar message [ 4407.828579] Lustre: Skipped 1 previous similar message [ 4409.581853] Lustre: server umount lustre-OST0001 complete [ 4412.925939] LustreError: 140294:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4412.947733] LustreError: 140294:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4415.457045] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4415.462691] LustreError: Skipped 1 previous similar message [ 4415.464742] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4415.509209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4415.517600] Lustre: Skipped 1 previous similar message [ 4418.042065] LustreError: 140294:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4418.057048] LustreError: 140294:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4419.450744] Lustre: server umount lustre-OST0000 complete [ 4428.258511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4428.265282] Lustre: Skipped 6 previous similar messages [ 4431.288970] Lustre: server umount lustre-MDT0000 complete [ 4433.379930] LustreError: 141498:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4433.405959] LustreError: 141498:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4435.177230] LustreError: 138451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789408054 with bad export cookie 7210815415460707847 [ 4435.192605] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4435.195351] LustreError: 138451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4435.559386] Lustre: server umount lustre-MDT0001 complete [ 4454.980725] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 4457.875993] Key type lgssc unregistered [ 4458.349952] LNet: 144850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4458.353777] LNetError: 144850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4458.367738] LNet: Removed LNI 192.168.202.106@tcp [ 4459.676749] Key type .llcrypt unregistered [ 4459.679300] Key type ._llcrypt unregistered [ 4505.639095] Key type ._llcrypt registered [ 4505.640755] Key type .llcrypt registered [ 4505.738295] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 4523.958139] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 4525.240135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4525.285160] alg: No test for adler32 (adler32-zlib) [ 4526.207710] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 4526.446996] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 4528.247252] Key type lgssc registered [ 4529.772066] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4566.311657] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4566.335081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4567.583163] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4567.622667] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4567.689217] Lustre: lustre-MDT0000: new disk, initializing [ 4567.749354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4567.759384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4573.608069] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4587.163427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4587.456759] Lustre: 149004:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4587.579388] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4587.586308] Lustre: Skipped 1 previous similar message [ 4587.752569] Lustre: lustre-MDT0001: new disk, initializing [ 4587.917784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4587.986348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4588.006025] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4595.477304] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4607.027154] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4612.942491] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4619.914572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4620.162654] Lustre: lustre-OST0000: new disk, initializing [ 4620.166928] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4620.172033] Lustre: 150975:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4620.330029] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4628.147586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4628.153919] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4628.314416] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4628.543214] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4639.136467] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4641.272361] LustreError: 149011:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255746 [ 4641.286886] LustreError: 149011:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255746 [ 4644.622877] Lustre: Failing over lustre-OST0000 [ 4644.844839] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4644.856155] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4644.978482] Lustre: server umount lustre-OST0000 complete [ 4664.720799] Lustre: 147226:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789408268/real 1789408268] req@00000000504df7a0 x1876330435088000/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789408284 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4664.764860] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4665.295171] Lustre: 147227:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789408268/real 1789408268] req@00000000facdcbb6 x1876330435088256/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789408284 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4666.227478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4666.893466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4666.930884] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4669.919126] Lustre: 147228:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789408273/real 1789408273] req@000000003610a3a3 x1876330435089280/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789408289 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4672.594261] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4674.285237] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4675.039499] Lustre: 147228:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789408278/real 1789408278] req@000000004f107804 x1876330435090304/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789408294 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4677.846245] Lustre: Failing over lustre-MDT0000 [ 4678.377854] Lustre: server umount lustre-MDT0000 complete [ 4680.159898] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4680.160851] Lustre: 147228:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789408283/real 1789408283] req@0000000071ed9b20 x1876330435091328/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789408299 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4680.165554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4680.166579] LustreError: 149015:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4680.166587] LustreError: 149015:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4682.848639] LustreError: 149011:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4682.859030] LustreError: 149011:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4685.285036] LustreError: 150187:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4685.307472] LustreError: 150187:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4687.955700] LustreError: 149010:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4693.065480] LustreError: 149009:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4693.096690] LustreError: 149009:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4696.543488] Lustre: 147227:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789408299/real 1789408299] req@000000005f985f4d x1876330435099392/t0(0) o400->MGC192.168.202.106@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789408315 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4696.576481] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4697.264557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4701.666393] LustreError: 149015:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4701.693896] LustreError: 149015:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 4705.761058] LustreError: 147224:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000654880b x1876330435103360/t0(0) o250->MGC192.168.202.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4706.023872] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 4706.057329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4706.107686] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4706.394671] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4710.703561] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4711.426577] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4711.450485] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4711.508970] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4721.632933] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4721.634319] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4721.637076] LustreError: Skipped 1 previous similar message [ 4721.641817] Lustre: Skipped 3 previous similar messages [ 4721.644899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4721.648188] Lustre: Skipped 1 previous similar message [ 4724.480834] Lustre: server umount lustre-OST0000 complete [ 4731.872895] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4731.876416] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4731.887557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4731.900127] Lustre: Skipped 1 previous similar message [ 4736.993466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4742.115465] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4742.118805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4742.124881] Lustre: Skipped 2 previous similar messages [ 4742.234705] Lustre: server umount lustre-MDT0000 complete [ 4745.295883] LustreError: 150972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789408364 with bad export cookie 1037694474042511643 [ 4745.310878] LustreError: 150972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4745.311554] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4745.915431] Lustre: server umount lustre-MDT0001 complete [ 4759.785762] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 4764.044755] Key type lgssc unregistered [ 4764.397883] LNet: 154627:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4764.406033] LNetError: 154627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4764.427865] LNet: Removed LNI 192.168.202.106@tcp [ 4765.856233] Key type .llcrypt unregistered [ 4765.863708] Key type ._llcrypt unregistered [ 4805.447720] Key type ._llcrypt registered [ 4805.449218] Key type .llcrypt registered [ 4805.538694] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 4806.801357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4806.857035] alg: No test for adler32 (adler32-zlib) [ 4807.850671] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 4808.095116] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 4809.775306] Key type lgssc registered [ 4810.716485] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4819.186585] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4819.211969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4820.797142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4825.587752] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4831.206664] LustreError: 156146:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4831.233812] LustreError: 156146:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4834.108442] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4836.323688] LustreError: 156145:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4839.789579] LustreError: 156174:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4839.820166] LustreError: 156174:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4839.846198] LustreError: 156174:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 4840.275126] Lustre: server umount lustre-MDT0000 complete [ 4857.870228] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 4868.995260] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 4907.321657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4907.657804] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4907.690114] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4907.801604] Lustre: lustre-MDT0000: new disk, initializing [ 4907.882728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4907.899475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4911.879237] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4923.849174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4923.979227] Lustre: 159477:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4924.046966] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4924.050059] Lustre: Skipped 1 previous similar message [ 4924.151234] Lustre: lustre-MDT0001: new disk, initializing [ 4924.198053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4924.229105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4924.243029] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4930.094271] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4940.865941] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4946.735646] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4953.936861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4954.142728] Lustre: lustre-OST0000: new disk, initializing [ 4954.145920] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4954.150852] Lustre: 161446:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4954.203746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4960.432352] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4960.806488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4960.812709] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4960.864536] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4969.042744] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4972.179527] LustreError: 159987:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255746 [ 4972.192518] LustreError: 159987:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255746 [ 4974.001746] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4976.164629] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 13:56:33 (1789408593) [ 4994.926926] LustreError: 159987:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255746 [ 4994.936157] LustreError: 159987:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255746 [ 4997.571591] LustreError: 159483:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023256130 [ 4997.576058] LustreError: 159483:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023256130 [ 5226.464933] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5226.476634] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5226.499180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5226.985389] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5227.005204] Lustre: Skipped 1 previous similar message [ 5227.010812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5227.019825] Lustre: Skipped 1 previous similar message [ 5232.096227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5232.099061] Lustre: Skipped 2 previous similar messages [ 5232.529243] Lustre: server umount lustre-MDT0000 complete [ 5237.071622] LustreError: 161445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789408856 with bad export cookie 1792489596796340898 [ 5237.076046] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5237.076406] LustreError: 161445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5237.216295] LustreError: 159987:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5237.219833] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5237.248654] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5242.337800] LustreError: 163214:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5242.339324] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5243.707838] Lustre: server umount lustre-MDT0001 complete [ 5254.737254] Lustre: server umount lustre-OST0000 complete [ 5266.030402] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 5278.513722] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 5318.466726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5318.916282] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5318.941721] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5319.018662] Lustre: lustre-MDT0000: new disk, initializing [ 5319.122570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5319.154106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5325.500639] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5338.827098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5338.944220] Lustre: 166120:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5338.988969] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5338.992761] Lustre: Skipped 1 previous similar message [ 5339.058316] Lustre: lustre-MDT0001: new disk, initializing [ 5339.150073] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5339.198857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5339.212224] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5344.788635] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5354.542160] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5360.450058] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5367.837718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5368.253087] Lustre: lustre-OST0000: new disk, initializing [ 5368.258082] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5368.267819] Lustre: 168087:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5368.441578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5376.170279] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5377.121767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5377.133824] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5377.342133] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5386.300657] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5391.548943] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5391.885167] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5398.425818] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5398.795947] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5403.108873] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5403.110326] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5403.115354] LustreError: Skipped 1 previous similar message [ 5403.116464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5403.149563] Lustre: Skipped 1 previous similar message [ 5407.671774] Lustre: server umount lustre-OST0000 complete [ 5413.345628] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5413.376150] Lustre: Skipped 1 previous similar message [ 5418.385224] Lustre: server umount lustre-MDT0000 complete [ 5418.477709] LustreError: 166131:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5418.501231] LustreError: 166131:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5422.717797] LustreError: 166112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789409042 with bad export cookie 1792489596796499728 [ 5422.725338] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5422.729989] LustreError: 166112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5423.231380] Lustre: server umount lustre-MDT0001 complete [ 5432.525807] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 14:04:10 (1789409050) [ 5451.691353] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 5463.103915] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 5503.166682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5503.492160] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5503.539221] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5503.621177] Lustre: lustre-MDT0000: new disk, initializing [ 5503.775957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5503.810514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5509.492840] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5521.534739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5521.692052] Lustre: 172847:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5522.036407] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5526.570677] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5536.518876] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5543.627174] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5552.505246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5552.924281] Lustre: lustre-OST0000: new disk, initializing [ 5552.927154] Lustre: Skipped 1 previous similar message [ 5552.930784] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5552.935265] Lustre: Skipped 2 previous similar messages [ 5552.939059] Lustre: 174826:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5558.637173] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5559.924551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5559.943872] Lustre: Skipped 1 previous similar message [ 5559.958835] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5560.074925] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5568.242128] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5569.066587] LustreError: 172856:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255746 [ 5569.081625] LustreError: 172856:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255746 [ 5580.773034] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5580.779867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5580.814915] Lustre: Skipped 1 previous similar message [ 5580.830449] Lustre: Skipped 4 previous similar messages [ 5583.330560] Lustre: server umount lustre-OST0000 complete [ 5588.448086] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5594.134338] Lustre: server umount lustre-MDT0000 complete [ 5596.138385] LustreError: 172856:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5596.157771] LustreError: 172856:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5597.667544] LustreError: 172839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789409216 with bad export cookie 1792489596796500603 [ 5597.676662] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5597.690101] LustreError: 172839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5598.121404] Lustre: server umount lustre-MDT0001 complete [ 5607.107336] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 14:07:04 (1789409224) [ 5626.295474] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 5637.042409] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 5677.406735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5677.739561] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5677.767912] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5677.852052] Lustre: lustre-MDT0000: new disk, initializing [ 5677.956093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5677.959688] Lustre: Skipped 2 previous similar messages [ 5677.974260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5684.717181] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5696.836196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5697.101916] Lustre: 179148:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5697.330706] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5702.336776] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5711.687055] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5717.920399] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5726.190770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5726.684017] Lustre: 181127:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5734.706843] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5735.528466] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5735.735729] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5747.193330] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5748.214720] LustreError: 180765:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255746 [ 5748.220722] LustreError: 180765:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255746 [ 5756.390260] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5756.398460] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5756.400832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5756.413956] Lustre: Skipped 3 previous similar messages [ 5756.429420] Lustre: Skipped 3 previous similar messages [ 5761.795485] Lustre: server umount lustre-OST0000 complete [ 5772.236584] Lustre: server umount lustre-MDT0000 complete [ 5776.232657] LustreError: 179140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789409395 with bad export cookie 1792489596796501695 [ 5776.254462] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5776.586608] Lustre: server umount lustre-MDT0001 complete [ 5787.830789] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 5790.321985] Key type lgssc unregistered [ 5790.663441] LNet: 182982:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5790.670465] LNetError: 182982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5790.688992] LNet: Removed LNI 192.168.202.106@tcp [ 5791.541566] Key type .llcrypt unregistered [ 5791.543734] Key type ._llcrypt unregistered [ 5812.747949] Key type ._llcrypt registered [ 5812.751399] Key type .llcrypt registered [ 5812.891969] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 5814.183204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5814.278201] alg: No test for adler32 (adler32-zlib) [ 5815.363423] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 5815.614771] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 5817.296360] Key type lgssc registered [ 5818.426593] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5826.484050] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5826.499842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5828.004962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5833.058142] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5838.306221] LustreError: 184389:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5838.322281] LustreError: 184389:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5842.395760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5842.777957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5847.474636] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5855.397583] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5861.138557] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5868.621283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5869.024143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5873.178397] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5876.667949] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5885.038224] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5890.136356] LustreError: 184384:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2207613190210 [ 5890.145320] LustreError: 184384:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2207613190210 [ 5907.938116] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5907.944083] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5907.967960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5907.981062] Lustre: Skipped 1 previous similar message [ 5911.880672] Lustre: server umount lustre-OST0000 complete [ 5918.178615] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5918.185129] Lustre: Skipped 1 previous similar message [ 5918.190721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5918.199734] Lustre: Skipped 1 previous similar message [ 5921.762031] Lustre: server umount lustre-MDT0000 complete [ 5923.300270] LustreError: 184389:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5924.815825] LustreError: 184368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789409544 with bad export cookie 13551800846267938513 [ 5924.841446] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5925.217808] Lustre: server umount lustre-MDT0001 complete [ 5938.211240] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 5941.996651] Key type lgssc unregistered [ 5942.284732] LNet: 187943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5942.288572] LNetError: 187943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5942.316009] LNet: Removed LNI 192.168.202.106@tcp [ 5943.259175] Key type .llcrypt unregistered [ 5943.263970] Key type ._llcrypt unregistered [ 5967.161378] Key type ._llcrypt registered [ 5967.163151] Key type .llcrypt registered [ 5967.243418] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 5968.227222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5968.255986] alg: No test for adler32 (adler32-zlib) [ 5969.608245] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 5969.984941] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 5971.647438] Key type lgssc registered [ 5972.992094] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5981.082996] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5981.108148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5982.683187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5987.052498] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5991.911243] LustreError: 189370:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5991.941621] LustreError: 189370:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5994.651397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5994.936158] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5998.995516] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6005.260932] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6012.373560] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6019.349279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6019.686364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6026.830694] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6026.956830] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6037.848525] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6043.675597] LustreError: 189366:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2207613190210 [ 6043.680229] LustreError: 189366:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2207613190210 [ 6046.688845] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6046.694767] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6046.719921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6047.718678] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6047.721164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6047.731348] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6047.769903] Lustre: Skipped 1 previous similar message [ 6051.451246] Lustre: lustre-OST0000: Not available for connect from 192.168.202.6@tcp (stopping) [ 6052.609814] Lustre: server umount lustre-OST0000 complete [ 6059.084910] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8c8284892000.ost_server_uuid 50 [ 6060.528630] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6077.753380] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6077.770796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6080.373688] Lustre: server umount lustre-MDT0000 complete [ 6084.070476] LustreError: 190099:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6084.093682] LustreError: 190099:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6086.278920] LustreError: 189350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789409705 with bad export cookie 5921663119798531574 [ 6086.285017] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6086.304923] LustreError: 189350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6086.965504] Lustre: server umount lustre-MDT0001 complete [ 6098.969760] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 14:15:16 (1789409716) [ 6107.622849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6108.395281] LustreError: 193290:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6108.532600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6113.148597] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6113.761638] LustreError: 193291:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6117.866940] LustreError: 193290:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6121.966841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6122.451550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6126.895090] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6134.670350] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6141.341497] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6148.328533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6148.858863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6153.980105] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6155.771223] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6163.436265] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6170.932087] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6171.154265] Lustre: lustre-OST0001: new disk, initializing [ 6171.156501] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6171.162450] Lustre: 196131:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6171.234156] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6177.164752] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6180.535048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6180.556585] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6180.742788] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6187.600104] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6188.403906] LustreError: 193285:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2207613190210 [ 6188.408212] LustreError: 193285:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2207613190210 [ 6191.085839] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6191.096765] Lustre: Skipped 1 previous similar message [ 6191.105876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6191.111615] Lustre: Skipped 3 previous similar messages [ 6196.686745] Lustre: server umount lustre-OST0000 complete [ 6198.344097] LustreError: 195111:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6203.004483] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6203.262822] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6208.994441] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6209.275931] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6216.678595] LustreError: 195113:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6216.698124] LustreError: 195113:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 6226.914496] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6226.918116] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6226.932323] LustreError: Skipped 1 previous similar message [ 6226.934665] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6226.941219] Lustre: Skipped 2 previous similar messages [ 6226.979120] Lustre: Skipped 4 previous similar messages [ 6230.977996] Lustre: server umount lustre-OST0001 complete [ 6237.153380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6237.182296] Lustre: Skipped 1 previous similar message [ 6242.512657] Lustre: server umount lustre-MDT0000 complete [ 6246.564214] LustreError: 193271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789409865 with bad export cookie 5921663119798532638 [ 6246.565772] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6246.573436] LustreError: 193271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6246.805584] Lustre: server umount lustre-MDT0001 complete [ 6272.551748] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 14:18:09 (1789409889) [ 6280.869687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6281.269160] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6281.320391] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6281.509750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6285.946822] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6295.901482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6295.973971] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6296.003442] Lustre: 199574:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6301.804598] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6310.368625] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6316.406575] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6323.837957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6324.085863] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6331.746875] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6332.636888] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6340.144596] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6346.770375] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6346.836396] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6347.045604] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6347.049375] Lustre: Skipped 2 previous similar messages [ 6352.657540] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6353.073050] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6359.727851] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6360.561781] LustreError: 199581:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2207613190210 [ 6360.565608] LustreError: 199581:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2207613190210 [ 6363.618091] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6363.627377] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6363.637196] LustreError: Skipped 1 previous similar message [ 6363.654643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6363.654652] Lustre: Skipped 4 previous similar messages [ 6368.756386] LustreError: 201426:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6368.782105] LustreError: 201426:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 6369.057599] Lustre: server umount lustre-OST0000 complete [ 6389.244577] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6389.255819] LustreError: Skipped 1 previous similar message [ 6391.545672] Lustre: server umount lustre-OST0001 complete [ 6398.944378] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6398.949099] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6398.962632] Lustre: Skipped 3 previous similar messages [ 6398.968956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6398.974856] Lustre: Skipped 4 previous similar messages [ 6402.002730] Lustre: server umount lustre-MDT0000 complete [ 6406.423430] LustreError: 202458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789410025 with bad export cookie 5921663119798534052 [ 6406.440936] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6406.443779] LustreError: 202458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6407.001229] Lustre: server umount lustre-MDT0001 complete [ 6432.758786] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 14:20:50 (1789410050) [ 6450.017151] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 6461.700529] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 6506.426494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6506.635970] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6506.661310] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6506.754400] Lustre: lustre-MDT0000: new disk, initializing [ 6506.830790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6506.870180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6512.082266] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6525.776038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6526.002913] Lustre: 207672:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6526.128440] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6526.132132] Lustre: Skipped 1 previous similar message [ 6526.203193] Lustre: lustre-MDT0001: new disk, initializing [ 6526.368148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6526.390683] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6533.418169] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6544.985808] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6550.355869] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6556.987960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6557.419184] Lustre: lustre-OST0000: new disk, initializing [ 6557.422601] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6557.428278] Lustre: 209641:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6565.472308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6565.486058] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6565.616279] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6566.066852] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6578.291416] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6584.686099] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6584.995854] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6591.855536] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6592.293501] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6596.579607] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6596.595064] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6596.604417] LustreError: Skipped 1 previous similar message [ 6596.612777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6596.626016] Lustre: Skipped 2 previous similar messages [ 6596.635656] Lustre: Skipped 2 previous similar messages [ 6609.375423] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6609.777067] Lustre: server umount lustre-OST0000 complete [ 6620.619994] Lustre: server umount lustre-MDT0000 complete [ 6622.177773] LustreError: 207676:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6622.187387] LustreError: 207676:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [ 6624.603704] LustreError: 207664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789410243 with bad export cookie 5921663119798535837 [ 6624.605667] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6624.612463] LustreError: 207664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6625.326674] Lustre: server umount lustre-MDT0001 complete [ 6635.205222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6640.983279] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6651.099643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6656.903849] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6666.641322] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6673.421158] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6682.250531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6690.909976] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6699.441402] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6706.031035] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6706.327105] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6711.267618] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6711.415350] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6715.874191] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6720.011488] Lustre: server umount lustre-OST0000 complete [ 6727.453674] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6732.657345] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6762.520618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6769.575886] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6778.606745] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6780.918587] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6781.234531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6781.261727] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6781.275100] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6781.288397] Lustre: Skipped 1 previous similar message [ 6786.532695] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6786.533927] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6786.558263] Lustre: Skipped 4 previous similar messages [ 6786.563225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6786.596259] Lustre: Skipped 10 previous similar messages [ 6792.710258] Lustre: server umount lustre-OST0000 complete [ 6811.103923] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6811.432511] Lustre: server umount lustre-MDT0000 complete [ 6812.131519] LustreError: 216596:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6812.165514] LustreError: 216596:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 6815.317240] LustreError: 211842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789410434 with bad export cookie 5921663119798536712 [ 6815.326522] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6815.488554] Lustre: server umount lustre-MDT0001 complete [ 6823.750659] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 14:27:21 (1789410441) [ 6832.443866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6833.131356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6833.138637] Lustre: Skipped 6 previous similar messages [ 6838.631251] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6848.247329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6853.508786] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6861.888944] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6867.928290] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6875.501400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6882.873857] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6890.893157] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6895.158852] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6895.314963] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6899.560133] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6899.781460] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6906.921643] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6907.143935] Lustre: lustre-OST0001: new disk, initializing [ 6907.147300] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6907.153710] Lustre: 221208:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6913.797503] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6915.753896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6915.764519] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6915.856724] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6923.226723] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6929.432792] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6929.668374] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6934.795224] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6934.994197] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6941.672942] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6941.695547] LustreError: Skipped 1 previous similar message [ 6943.481623] Lustre: server umount lustre-OST0001 complete [ 6952.387525] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6952.781061] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6960.446768] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6960.685144] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6992.787518] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6998.523465] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6998.548491] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6998.548761] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6998.574607] Lustre: Skipped 1 previous similar message [ 6999.634380] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7007.320342] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7008.555952] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7016.144110] Lustre: server umount lustre-OST0001 complete [ 7044.579732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7044.588509] Lustre: Skipped 18 previous similar messages [ 7050.207133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7050.559952] Lustre: server umount lustre-OST0000 complete [ 7058.400186] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7058.408652] Lustre: Skipped 9 previous similar messages [ 7061.482537] Lustre: server umount lustre-MDT0000 complete [ 7064.740871] LustreError: 217816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789410684 with bad export cookie 5921663119798537685 [ 7064.751619] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7064.755847] LustreError: 217816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7079.391121] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7079.759138] Lustre: server umount lustre-MDT0001 complete [ 7104.131843] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 14:32:02 (1789410722) [ 7111.169298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7111.644190] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7111.711181] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7117.805550] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7127.192826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7127.328159] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7127.344915] Lustre: 226454:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7132.909518] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7140.843366] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7146.140159] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7153.145803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7153.451064] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7160.668911] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7170.001391] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7182.178233] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7182.251370] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7190.885155] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7198.308826] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7199.956613] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7204.061977] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7204.441089] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7209.070497] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7209.249421] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7214.613438] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8c829174e000.ost_server_uuid 50 [ 7216.707971] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8c829174e000.ost_server_uuid in IDLE state after 0 sec [ 7216.844940] LustreError: 226462:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199040032834 [ 7216.854329] LustreError: 226462:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199040032834 [ 7217.785888] Lustre: Permanently deactivating lustre-OST0001 [ 7217.804718] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7221.723333] Lustre: Permanently reactivating lustre-OST0001 [ 7225.826500] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7225.836586] LustreError: Skipped 6 previous similar messages [ 7231.969397] Lustre: server umount lustre-OST0001 complete [ 7236.580763] LustreError: 229334:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7236.604493] LustreError: 229334:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 46 previous similar messages [ 7242.758866] Lustre: server umount lustre-OST0000 complete [ 7254.030580] Lustre: server umount lustre-MDT0000 complete [ 7257.985050] LustreError: 226446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789410877 with bad export cookie 5921663119798538973 [ 7257.995088] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7257.999649] LustreError: 226446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7258.321376] Lustre: server umount lustre-MDT0001 complete [ 7284.374812] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 14:35:01 (1789410901) [ 7292.261821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7292.597952] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7292.622692] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7292.627972] Lustre: Skipped 5 previous similar messages [ 7296.981399] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7305.939357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7306.071882] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7306.091163] Lustre: 233153:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7311.605352] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7319.800189] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7325.132107] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7332.856345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7333.101812] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7333.160814] Lustre: Permanently deactivating lustre-OST0000 [ 7333.202423] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7340.554235] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7341.641134] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7349.077688] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7356.767808] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7356.895177] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7357.164244] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7357.167876] Lustre: Skipped 11 previous similar messages [ 7364.467872] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7365.302833] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7374.203562] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7376.281887] Lustre: Permanently reactivating lustre-OST0000 [ 7376.310064] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7376.317412] Lustre: Skipped 2 previous similar messages [ 7382.612442] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 40 seconds. I think it's dead, and I am evicting it. exp 00000000f4dafa29, cur 1789411001 deadline 1789410991 last 1789410961 [ 7382.612540] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7382.624569] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7382.630911] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7382.675881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7384.762883] LustreError: 233160:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023256130 [ 7384.769798] LustreError: 233160:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023256130 [ 7384.776343] LustreError: 233160:0:(mdt_open.c:646:mdt_finish_open()) Skipped 1 previous similar message [ 7397.582965] Lustre: server umount lustre-OST0001 complete [ 7408.423572] Lustre: server umount lustre-OST0000 complete [ 7419.530593] Lustre: server umount lustre-MDT0000 complete [ 7423.257550] LustreError: 233143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789411042 with bad export cookie 5921663119798540653 [ 7423.262213] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7423.264757] LustreError: 233143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7423.505658] Lustre: server umount lustre-MDT0001 complete [ 7439.262722] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 7442.176636] Key type lgssc unregistered [ 7442.393602] LNet: 238033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7442.397782] LNetError: 238033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7442.417474] LNet: Removed LNI 192.168.202.106@tcp [ 7443.206729] Key type .llcrypt unregistered [ 7443.210937] Key type ._llcrypt unregistered [ 7470.527882] Key type ._llcrypt registered [ 7470.531607] Key type .llcrypt registered [ 7470.697601] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 7472.251955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7472.466276] alg: No test for adler32 (adler32-zlib) [ 7474.024790] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 7474.398574] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 7476.295288] Key type lgssc registered [ 7478.022967] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7493.206757] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7493.248560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7495.193208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7502.260623] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7505.386687] LustreError: 239498:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7505.437566] LustreError: 239498:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7510.497670] LustreError: 239499:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7513.055585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7513.280798] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7513.329740] Lustre: 239488:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7513.343601] Lustre: Permanently deactivating lustre-MDT0001 [ 7513.370438] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7513.753558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7519.296677] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7519.381530] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7528.310981] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7534.203333] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7541.835498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7542.405932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7549.241773] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7555.388397] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7557.731239] LustreError: 241341:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7557.750675] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7563.238867] LustreError: 242063:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7563.773231] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7564.081363] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7565.167519] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7565.192672] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7565.209136] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7572.109934] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7580.528359] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7583.985774] Lustre: Permanently reactivating lustre-MDT0001 [ 7584.003406] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7584.012530] Lustre: Skipped 2 previous similar messages [ 7587.374284] Lustre: lustre-MDT0001: Client a5ef2673-a476-4d1d-bbb5-21798faa3676 (at 192.168.202.6@tcp) reconnecting [ 7590.566979] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7590.575754] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7590.586334] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7590.600019] LustreError: 243211:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7590.603394] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7590.613977] LustreError: 243211:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7590.614122] LustreError: 243211:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.106@tcp: cfg command failed: rc = -108 [ 7590.615917] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7590.615917] [ 7590.616811] LustreError: 239491:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7593.314499] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7596.623697] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7605.634754] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7605.950711] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7612.565674] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7612.890178] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7615.180953] LustreError: 241003:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255619 [ 7615.185550] LustreError: 241003:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255619 [ 7616.694704] Lustre: Permanently deactivating lustre-MDT0001 [ 7616.717350] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7616.722274] Lustre: Skipped 2 previous similar messages [ 7625.893049] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7632.107598] LustreError: 239493:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2207613255681 [ 7636.448876] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7636.461449] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7636.470480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7636.972719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7636.977844] Lustre: Skipped 2 previous similar messages [ 7641.805688] Lustre: server umount lustre-MDT0000 complete [ 7642.086892] LustreError: 239493:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7642.108161] LustreError: 239493:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7646.299152] LustreError: 243987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789411265 with bad export cookie 8026418002762237128 [ 7646.310259] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7646.325333] LustreError: 243987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7646.882419] Lustre: server umount lustre-MDT0001 complete [ 7653.334418] Lustre: server umount lustre-OST0000 complete [ 7669.009788] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 14:41:26 (1789411286) [ 7682.131057] Lustre: server umount lustre-OST0001 complete [ 7689.016763] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 7700.225787] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 7739.089517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7739.344417] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7739.358635] Lustre: Skipped 2 previous similar messages [ 7739.403345] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7739.500040] Lustre: lustre-MDT0000: new disk, initializing [ 7739.566274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7739.597306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7745.427658] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7756.598587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7756.698209] Lustre: 247811:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7756.741513] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7756.747494] Lustre: Skipped 1 previous similar message [ 7756.803951] Lustre: lustre-MDT0001: new disk, initializing [ 7756.876960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7756.931785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7756.950694] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7762.283314] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7771.759800] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7777.455729] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7785.909100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7786.169171] Lustre: lustre-OST0000: new disk, initializing [ 7786.173573] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7786.189235] Lustre: 249784:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7786.303307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7793.278700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7793.293193] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7793.438438] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7793.798658] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7807.472737] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7808.467311] LustreError: 247820:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255746 [ 7808.470995] LustreError: 247820:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 1 previous similar message [ 7808.475350] LustreError: 247820:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255746 [ 7809.653306] LustreError: 247821:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7818.697973] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7818.911166] Lustre: lustre-OST0001: new disk, initializing [ 7818.913695] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7818.919478] Lustre: 250900:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7819.002468] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7819.676284] LustreError: 247821:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7819.694643] LustreError: 247821:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2207613190210 [ 7819.709475] LustreError: 247821:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 2 previous similar messages [ 7819.731142] LustreError: 247821:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2207613190210 [ 7819.793215] LustreError: 248678:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7826.252813] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7829.051877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7829.058093] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7829.134066] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7829.815204] LustreError: 248678:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7837.197850] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7839.040811] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7844.328533] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7844.338598] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7844.357703] Lustre: Skipped 4 previous similar messages [ 7847.598620] Lustre: server umount lustre-OST0001 complete [ 7849.451169] LustreError: 249767:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7849.469073] LustreError: 249767:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7864.803803] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7864.822947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7864.826940] Lustre: Skipped 1 previous similar message [ 7869.931055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7869.937465] LustreError: 249770:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7869.942238] Lustre: Skipped 1 previous similar message [ 7869.962899] LustreError: 249770:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 7870.928038] Lustre: server umount lustre-OST0000 complete [ 7874.529457] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7874.536600] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7874.544761] Lustre: Skipped 1 previous similar message [ 7880.164538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7880.171850] Lustre: Skipped 4 previous similar messages [ 7880.718214] Lustre: server umount lustre-MDT0000 complete [ 7884.486415] LustreError: 247805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789411503 with bad export cookie 8026418002762239942 [ 7884.490788] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7884.495023] LustreError: 247805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7890.931760] Lustre: server umount lustre-MDT0001 complete [ 7902.545254] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 7905.133614] Key type lgssc unregistered [ 7905.309323] LNet: 252935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7905.314502] LNetError: 252935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7905.334437] LNet: Removed LNI 192.168.202.106@tcp [ 7905.875586] Key type .llcrypt unregistered [ 7905.879441] Key type ._llcrypt unregistered [ 7950.711537] Key type ._llcrypt registered [ 7950.713638] Key type .llcrypt registered [ 7950.791669] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 7951.991744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7952.085827] alg: No test for adler32 (adler32-zlib) [ 7953.192190] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 7953.528840] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 7955.239255] Key type lgssc registered [ 7956.406247] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7964.643816] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7964.658434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7966.073197] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7966.117983] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7966.376705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7970.719941] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7980.120156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7980.378201] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7980.459144] Lustre: 255132:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7981.046735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7985.903147] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7995.338678] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8000.490186] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8007.336659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8007.520636] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8007.742776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8014.125224] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8015.145383] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8023.498801] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8028.513425] LustreError: 255138:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199040032834 [ 8028.599985] LustreError: 255139:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255619 [ 8029.919551] LustreError: 255140:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255619 [ 8029.924163] LustreError: 255140:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 2 previous similar messages [ 8029.928395] LustreError: 255140:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255619 [ 8031.265381] LustreError: 255138:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255619 [ 8031.269951] LustreError: 255138:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255619 [ 8033.470744] LustreError: 255138:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255619 [ 8033.474836] LustreError: 255138:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 2 previous similar messages [ 8033.478788] LustreError: 255138:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255619 [ 8033.482498] LustreError: 255138:0:(mdt_open.c:646:mdt_finish_open()) Skipped 2 previous similar messages [ 8037.766670] LustreError: 255139:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2207655133185 [ 8037.772282] LustreError: 255139:0:(mdt_open.c:646:mdt_finish_open()) Skipped 2 previous similar messages [ 8075.759737] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8075.761581] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8075.768503] LustreError: Skipped 1 previous similar message [ 8075.769519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8075.792749] Lustre: Skipped 1 previous similar message [ 8080.873716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8080.891327] Lustre: Skipped 1 previous similar message [ 8081.277975] Lustre: server umount lustre-OST0000 complete [ 8087.270808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8105.403121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8105.707296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8106.735859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8106.749595] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8106.751974] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8106.762729] Lustre: Skipped 1 previous similar message [ 8113.057696] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8120.720747] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8122.534087] LustreError: 255140:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2207655133185 [ 8122.545220] LustreError: 255140:0:(mdt_open.c:646:mdt_finish_open()) Skipped 7 previous similar messages [ 8163.299214] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8163.330911] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8163.334242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8163.357907] Lustre: Skipped 1 previous similar message [ 8166.695294] Lustre: server umount lustre-OST0000 complete [ 8173.549424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8173.554634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8173.571767] Lustre: Skipped 2 previous similar messages [ 8173.595193] Lustre: Skipped 1 previous similar message [ 8177.768564] Lustre: server umount lustre-MDT0000 complete [ 8181.898075] LustreError: 255123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789411801 with bad export cookie 17613458519688459075 [ 8181.911132] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8181.918842] LustreError: 255123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8182.263966] Lustre: server umount lustre-MDT0001 complete [ 8193.952300] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 8196.991671] Key type lgssc unregistered [ 8197.425329] LNet: 260558:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8197.432695] LNetError: 260558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8198.508622] LNet: Removed LNI 192.168.202.106@tcp [ 8199.689777] Key type .llcrypt unregistered [ 8199.692950] Key type ._llcrypt unregistered [ 8229.098418] Key type ._llcrypt registered [ 8229.100501] Key type .llcrypt registered [ 8229.299965] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 8230.483735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8230.500770] alg: No test for adler32 (adler32-zlib) [ 8231.976254] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 8232.326557] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 8234.039392] Key type lgssc registered [ 8234.698714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8242.625862] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8242.645834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8244.149530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8249.043262] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8254.433681] LustreError: 261971:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8254.462818] LustreError: 261971:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8258.176027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8258.568753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8263.715502] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8272.865784] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8281.931114] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8290.683686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8291.089931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8298.261801] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8306.322400] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8314.624366] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8321.818602] LustreError: 261966:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255746 [ 8321.824191] LustreError: 261966:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255746 [ 8339.431586] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8339.448353] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8339.461662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8340.462272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8340.481982] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8344.887622] Lustre: server umount lustre-OST0000 complete [ 8350.692702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8350.729617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8350.738316] Lustre: Skipped 2 previous similar messages [ 8355.811274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8355.818364] Lustre: Skipped 1 previous similar message [ 8356.590492] Lustre: server umount lustre-MDT0000 complete [ 8360.386495] LustreError: 261951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789411979 with bad export cookie 8221307212577892513 [ 8360.402643] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8360.413457] LustreError: 261951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8360.895867] Lustre: server umount lustre-MDT0001 complete [ 8372.472172] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 8376.710470] Key type lgssc unregistered [ 8376.997617] LNet: 266298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8377.015962] LNetError: 266298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8377.033398] LNet: Removed LNI 192.168.202.106@tcp [ 8378.291062] Key type .llcrypt unregistered [ 8378.294450] Key type ._llcrypt unregistered [ 8399.445095] Key type ._llcrypt registered [ 8399.452899] Key type .llcrypt registered [ 8399.597454] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 8400.988946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8401.120985] alg: No test for adler32 (adler32-zlib) [ 8402.305949] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 8402.541963] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 8404.239197] Key type lgssc registered [ 8405.815964] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8413.553198] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8413.569505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8414.859200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8419.978600] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8425.448965] LustreError: 267517:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8425.486514] LustreError: 267517:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8427.923251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8428.183664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8432.460294] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8443.944336] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8452.397565] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8463.667368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8464.319173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8472.205777] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8482.063030] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8484.810212] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8490.693126] LustreError: 267514:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255746 [ 8490.705533] LustreError: 267514:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255746 [ 8499.169745] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8499.183268] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8499.197403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8500.195779] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8500.203120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8500.222642] Lustre: Skipped 1 previous similar message [ 8501.104969] Lustre: server umount lustre-OST0000 complete [ 8505.312536] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8505.313728] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8505.328263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8505.342122] Lustre: Skipped 1 previous similar message [ 8505.360027] Lustre: Skipped 1 previous similar message [ 8510.433178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8511.810139] Lustre: server umount lustre-MDT0000 complete [ 8515.511238] LustreError: 267497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789412134 with bad export cookie 11903730670606574199 [ 8515.513119] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8515.525180] LustreError: 267497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8515.999793] Lustre: server umount lustre-MDT0001 complete [ 8528.652813] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 8531.886891] Key type lgssc unregistered [ 8532.164596] LNet: 271197:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8532.171647] LNetError: 271197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8532.193471] LNet: Removed LNI 192.168.202.106@tcp [ 8533.423188] Key type .llcrypt unregistered [ 8533.429923] Key type ._llcrypt unregistered [ 8558.825836] Key type ._llcrypt registered [ 8558.827919] Key type .llcrypt registered [ 8558.926294] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 8560.463463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8560.549870] alg: No test for adler32 (adler32-zlib) [ 8561.688370] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 8562.026409] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 8563.751266] Key type lgssc registered [ 8564.958506] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8573.596729] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8573.629492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8574.927760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8580.071162] LustreError: 272614:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8580.108736] LustreError: 272614:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8580.432251] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8585.215579] LustreError: 272615:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8590.315537] LustreError: 272614:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8592.014834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8592.768859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8597.120189] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8605.265864] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8611.083217] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8616.847648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8617.065284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8620.169032] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8624.688537] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8635.135557] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8641.393227] LustreError: 274825:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255746 [ 8641.398551] LustreError: 274825:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255746 [ 8661.486937] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8661.497256] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8661.519422] Lustre: Skipped 1 previous similar message [ 8661.528178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8666.594825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8666.603173] Lustre: Skipped 2 previous similar messages [ 8666.871734] Lustre: server umount lustre-OST0000 complete [ 8671.715288] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8671.758291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8671.779654] Lustre: Skipped 1 previous similar message [ 8676.881120] LustreError: 275060:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8676.908090] LustreError: 275060:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8676.958819] Lustre: server umount lustre-MDT0000 complete [ 8681.677449] LustreError: 273348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789412300 with bad export cookie 13923395190451694223 [ 8681.685540] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8682.077728] Lustre: server umount lustre-MDT0001 complete [ 8696.750312] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 8700.298585] Key type lgssc unregistered [ 8700.545784] LNet: 276924:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8700.549860] LNetError: 276924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8700.562186] LNet: Removed LNI 192.168.202.106@tcp [ 8701.485219] Key type .llcrypt unregistered [ 8701.486977] Key type ._llcrypt unregistered [ 8723.181549] Key type ._llcrypt registered [ 8723.192816] Key type .llcrypt registered [ 8723.441233] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 8725.245050] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8725.570267] alg: No test for adler32 (adler32-zlib) [ 8726.854542] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 8727.192031] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 8728.944118] Key type lgssc registered [ 8730.200724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8739.368805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8739.403356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8741.116529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8747.639936] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8751.587324] LustreError: 278160:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8751.616520] LustreError: 278160:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8756.711703] LustreError: 278162:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8760.343387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8760.813099] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8768.409542] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8778.813567] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8784.087806] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8791.517512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8792.086317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8801.834811] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8812.531168] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8812.563991] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8816.396822] LustreError: 278154:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255746 [ 8816.405810] LustreError: 278154:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255746 [ 8825.615717] Lustre: server umount lustre-OST0000 complete [ 8832.490978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8832.504954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8832.523900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8838.114728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8838.129951] Lustre: Skipped 2 previous similar messages [ 8840.356911] Lustre: server umount lustre-MDT0000 complete [ 8843.231421] Lustre: 277457:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789412446/real 1789412446] req@00000000ab61f50c x1876334840149632/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789412462 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8843.248419] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8843.253334] LustreError: 278166:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8843.256625] Lustre: Skipped 1 previous similar message [ 8843.270218] LustreError: 278166:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8844.268479] Lustre: 277454:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789412447/real 1789412447] req@00000000c14b206d x1876334840150400/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789412463 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8844.641754] LustreError: 280821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789412463 with bad export cookie 17004983693477337954 [ 8844.648829] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8844.651033] LustreError: 280821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8844.930357] Lustre: server umount lustre-MDT0001 complete [ 8860.214357] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 8865.376381] Key type lgssc unregistered [ 8865.905296] LNet: 281871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8865.913750] LNetError: 281871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8865.946886] LNet: Removed LNI 192.168.202.106@tcp [ 8867.354545] Key type .llcrypt unregistered [ 8867.358903] Key type ._llcrypt unregistered [ 8881.784876] Key type ._llcrypt registered [ 8881.787102] Key type .llcrypt registered [ 8881.908641] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8902.153408] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 8919.910929] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 8921.348668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8921.365568] alg: No test for adler32 (adler32-zlib) [ 8922.424521] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 8922.641506] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 8924.297118] Key type lgssc registered [ 8925.550932] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8965.725986] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8965.747274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8967.206136] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8967.230185] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8967.283934] Lustre: lustre-MDT0000: new disk, initializing [ 8967.339741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8967.356289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8971.788529] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8982.684388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8982.892318] Lustre: 285435:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8983.071070] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8983.077555] Lustre: Skipped 1 previous similar message [ 8983.238482] Lustre: lustre-MDT0001: new disk, initializing [ 8983.441683] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8983.517229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8983.522879] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8991.472935] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9002.466898] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9008.340502] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9018.867318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9019.282394] Lustre: lustre-OST0000: new disk, initializing [ 9019.286222] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9019.293302] Lustre: 287405:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9019.445698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9027.475725] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9028.183160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9028.205001] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9028.414545] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9038.496149] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9042.758447] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9042.920411] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9049.374924] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9049.741126] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9054.177039] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9054.191054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9054.192632] LustreError: Skipped 1 previous similar message [ 9054.205699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9054.218926] Lustre: Skipped 1 previous similar message [ 9054.265919] Lustre: Skipped 1 previous similar message [ 9059.219423] Lustre: server umount lustre-OST0000 complete [ 9064.418465] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9064.419932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9064.440179] Lustre: Skipped 1 previous similar message [ 9064.469754] Lustre: Skipped 1 previous similar message [ 9069.546512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9069.559767] Lustre: Skipped 1 previous similar message [ 9074.662911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9074.666842] Lustre: Skipped 1 previous similar message [ 9078.239154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9078.491671] Lustre: server umount lustre-MDT0000 complete [ 9083.014810] LustreError: 285428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789412702 with bad export cookie 4130647343051924874 [ 9083.022824] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9083.029233] LustreError: 285428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9083.401111] Lustre: server umount lustre-MDT0001 complete [ 9093.681765] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 15:05:11 (1789412711) [ 9101.645201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9102.235610] LustreError: 289821:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9102.271966] LustreError: 289821:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9102.416228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9107.428661] LustreError: 289822:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9107.939604] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9112.546754] LustreError: 289821:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9117.090519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9117.370672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9122.650253] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9132.118932] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9138.880912] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9147.318080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9147.889941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9154.948843] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9164.116648] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9178.794983] LustreError: 292312:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023256130 [ 9178.800597] LustreError: 292312:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023256130 [ 9179.297749] LustreError: 292312:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023256130 [ 9179.303760] LustreError: 292312:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 1 previous similar message [ 9179.308898] LustreError: 292312:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023256130 [ 9179.311641] LustreError: 292312:0:(mdt_open.c:646:mdt_finish_open()) Skipped 1 previous similar message [ 9187.812183] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9187.820105] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9187.829158] LustreError: Skipped 1 previous similar message [ 9187.835234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9187.847890] Lustre: Skipped 1 previous similar message [ 9191.687784] Lustre: server umount lustre-OST0000 complete [ 9198.050630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9198.068180] Lustre: Skipped 1 previous similar message [ 9198.082513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9198.088858] Lustre: Skipped 1 previous similar message [ 9203.024825] Lustre: server umount lustre-MDT0000 complete [ 9203.178207] LustreError: 289818:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9203.199449] LustreError: 289818:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9207.404437] LustreError: 289803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789412826 with bad export cookie 4130647343051925756 [ 9207.407181] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9207.413928] LustreError: 289803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9207.789931] Lustre: server umount lustre-MDT0001 complete [ 9220.568309] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 9223.773428] Key type lgssc unregistered [ 9224.099614] LNet: 293377:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9224.126422] LNetError: 293377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9225.191743] LNet: Removed LNI 192.168.202.106@tcp [ 9226.368660] Key type .llcrypt unregistered [ 9226.371042] Key type ._llcrypt unregistered [ 9257.337983] Key type ._llcrypt registered [ 9257.340297] Key type .llcrypt registered [ 9257.449818] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 9275.660928] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 9277.594301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9277.609168] alg: No test for adler32 (adler32-zlib) [ 9278.766571] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 9279.160018] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 9280.887245] Key type lgssc registered [ 9282.051255] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9336.240080] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9336.296318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9337.658903] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9337.683689] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9337.741610] Lustre: lustre-MDT0000: new disk, initializing [ 9337.807332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9337.821798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9343.179874] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9355.998553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9356.117834] Lustre: 297210:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9356.190299] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9356.199048] Lustre: Skipped 1 previous similar message [ 9356.290283] Lustre: lustre-MDT0001: new disk, initializing [ 9356.407261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9356.478474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9356.496143] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9362.251813] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9373.667663] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9380.630697] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9387.766165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9387.934129] Lustre: lustre-OST2710: new disk, initializing [ 9387.935912] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9387.939731] Lustre: 299181:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9387.987535] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9396.622879] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9398.365893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9398.373418] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9398.454947] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9409.234792] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9416.939628] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9417.130604] Lustre: lustre-OST03e8: new disk, initializing [ 9417.188652] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9417.201954] Lustre: 300245:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9417.339174] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9424.060988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9424.096631] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9424.292467] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9424.635211] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9432.105843] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9435.371541] LustreError: 298435:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255619 [ 9435.378273] LustreError: 298435:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255619 [ 9441.815637] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9442.064588] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9447.885421] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9448.155627] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9453.765558] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9454.031951] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9459.171454] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9459.479566] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9459.723297] LustreError: 297215:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199040032834 [ 9459.739593] LustreError: 297215:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199040032834 [ 9468.897576] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9468.913264] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9468.943960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9470.437122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9470.439484] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9472.402424] Lustre: server umount lustre-MDT0000 complete [ 9477.254193] LustreError: 299180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789413096 with bad export cookie 3668541664984807235 [ 9477.276350] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9477.285180] LustreError: 299180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9478.012517] Lustre: server umount lustre-MDT0001 complete [ 9492.297658] Lustre: server umount lustre-OST2710 complete [ 9496.031300] Lustre: 295004:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789413099/real 1789413099] req@000000002b67ef51 x1876335419007488/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789413115 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9496.060801] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9496.073521] Lustre: Skipped 2 previous similar messages [ 9497.176958] Lustre: server umount lustre-OST03e8 complete [ 9519.144634] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 9531.586671] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 9578.643732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9578.872487] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9578.903909] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9578.995544] Lustre: lustre-MDT0000: new disk, initializing [ 9579.127867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9579.139313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9584.166520] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9596.324555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9596.412516] Lustre: 305499:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9596.493807] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9596.497358] Lustre: Skipped 1 previous similar message [ 9596.570927] Lustre: lustre-MDT0001: new disk, initializing [ 9596.726412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9596.791360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9596.799371] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9603.745615] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9617.005398] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9624.230405] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9632.510751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9632.892988] Lustre: lustre-OST0000: new disk, initializing [ 9632.907721] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9632.927300] Lustre: 307469:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9633.120215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9638.502098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9638.513504] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9638.699335] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9641.823690] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9653.922850] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9659.840882] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9660.018722] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9665.755979] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9665.954374] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9669.603522] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9669.605550] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9669.616282] LustreError: Skipped 1 previous similar message [ 9669.617086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9669.617090] Lustre: Skipped 3 previous similar messages [ 9669.649314] Lustre: Skipped 1 previous similar message [ 9674.422535] Lustre: server umount lustre-OST0000 complete [ 9678.819248] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9678.834698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9678.849220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9678.864712] Lustre: Skipped 1 previous similar message [ 9684.962074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9684.966490] Lustre: Skipped 3 previous similar messages [ 9692.127321] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9692.406795] Lustre: server umount lustre-MDT0000 complete [ 9695.200435] LustreError: 305512:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9695.225162] LustreError: 305512:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 9696.827597] LustreError: 308810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789413316 with bad export cookie 3668541664984808922 [ 9696.832353] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9696.839249] LustreError: 308810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9697.651684] Lustre: server umount lustre-MDT0001 complete [ 9707.610227] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 15:15:25 (1789413325) [ 9709.276836] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9711.359660] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 15:15:29 (1789413329) [ 9715.569775] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 9748.676332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9749.005614] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9749.049316] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9749.371720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9755.241473] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9765.027731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9765.146860] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9765.207225] Lustre: 311329:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9770.293194] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9792.618379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9792.957977] Lustre: Denying initial registration attempt from nid 192.168.202.106@tcp, specified as failover [ 9792.969166] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9792.984470] LustreError: 312784:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -99 [ 9792.994880] LustreError: 312784:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 9793.004016] LustreError: 312784:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9793.132850] Lustre: server umount lustre-OST0000 complete [ 9793.134235] LustreError: 312784:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9801.184393] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9801.187825] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9801.205237] Lustre: Skipped 1 previous similar message [ 9801.209496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9801.213035] Lustre: Skipped 2 previous similar messages [ 9807.180374] Lustre: server umount lustre-MDT0000 complete [ 9809.889859] LustreError: 312079:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9809.915408] LustreError: 312079:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9811.004041] LustreError: 311324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789413430 with bad export cookie 3668541664984809811 [ 9811.010048] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9811.020944] LustreError: 311324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9811.286079] Lustre: server umount lustre-MDT0001 complete [ 9824.232784] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 9828.082771] Key type lgssc unregistered [ 9828.463585] LNet: 313903:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9828.472712] LNetError: 313903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9829.546265] LNet: Removed LNI 192.168.202.106@tcp [ 9830.710339] Key type .llcrypt unregistered [ 9830.714414] Key type ._llcrypt unregistered [ 9846.346480] Key type ._llcrypt registered [ 9846.349659] Key type .llcrypt registered [ 9846.527689] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 9848.378776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9848.439590] alg: No test for adler32 (adler32-zlib) [ 9849.784630] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 9850.418342] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 9852.280761] Key type lgssc registered [ 9853.986018] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9910.943815] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 9922.358946] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9922.376890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9923.637374] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9923.658253] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9923.873371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9929.672684] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9939.773479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9939.841882] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9939.908801] Lustre: 316667:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9940.226842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9945.723047] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9954.875667] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9960.948808] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9981.674896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9982.050101] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9982.202662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9989.085507] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9996.691427] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9999.844625] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9999.848517] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9999.865078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10000.359252] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10000.376091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10000.381252] Lustre: Skipped 1 previous similar message [10004.798278] Lustre: server umount lustre-OST0000 complete [10010.596124] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10010.635396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10010.660676] Lustre: Skipped 1 previous similar message [10015.725889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10015.741799] Lustre: Skipped 1 previous similar message [10015.944572] Lustre: server umount lustre-MDT0000 complete [10020.833934] LustreError: 316659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789413640 with bad export cookie 6194962288118384809 [10020.849955] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10020.860375] LustreError: 316659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10020.883966] LustreError: 314556:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000079f63979 x1876336017707136/t0(0) o38->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10021.604787] Lustre: server umount lustre-MDT0001 complete [10036.328156] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [10041.754389] Key type lgssc unregistered [10042.047454] LNet: 320241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10042.052723] LNetError: 320241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10042.063826] LNet: Removed LNI 192.168.202.106@tcp [10043.019837] Key type .llcrypt unregistered [10043.023416] Key type ._llcrypt unregistered [10073.825697] Key type ._llcrypt registered [10073.827711] Key type .llcrypt registered [10073.914674] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [10075.024746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10075.101575] alg: No test for adler32 (adler32-zlib) [10076.285790] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [10076.723681] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [10078.584234] Key type lgssc registered [10079.874861] Lustre: Echo OBD driver; http://www.lustre.org/ [10088.936907] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10088.965332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10090.492686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10095.802973] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10100.710223] LustreError: 321661:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10100.722919] LustreError: 321661:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10105.455732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10105.963075] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10110.655625] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10119.320187] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10125.535880] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10135.143110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10135.458852] LustreError: 323480:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10135.647390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10143.283829] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10154.432887] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10157.755674] LustreError: 321655:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255619 [10157.761167] LustreError: 321655:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255619 [10158.268113] LustreError: 321657:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255619 [10158.273433] LustreError: 321657:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 20 previous similar messages [10158.277659] LustreError: 321657:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255619 [10158.283196] LustreError: 321657:0:(mdt_open.c:646:mdt_finish_open()) Skipped 20 previous similar messages [10159.271385] LustreError: 321655:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255619 [10159.276462] LustreError: 321655:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 36 previous similar messages [10159.281367] LustreError: 321655:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255619 [10159.284766] LustreError: 321655:0:(mdt_open.c:646:mdt_finish_open()) Skipped 36 previous similar messages [10166.248364] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10166.264505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10166.281758] Lustre: Skipped 1 previous similar message [10167.714730] Lustre: server umount lustre-MDT0000 complete [10167.943514] LustreError: 321656:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10171.370570] LustreError: 321660:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10171.396926] LustreError: 321660:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10172.774391] LustreError: 321642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789413792 with bad export cookie 9371323216273530018 [10172.780691] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10172.785490] LustreError: 321642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10173.032327] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.6@tcp (stopping) [10176.481797] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10176.482137] LustreError: 321655:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10176.500815] Lustre: Skipped 2 previous similar messages [10176.502673] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10176.515547] LustreError: 321655:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10182.624984] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10182.635229] Lustre: Skipped 2 previous similar messages [10187.231160] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10187.558521] Lustre: server umount lustre-MDT0001 complete [10194.852663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10204.763535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10218.656219] LustreError: 323491:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10218.672702] LustreError: 323491:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [10218.735346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10218.795858] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10223.593561] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10232.004326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10232.191455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10237.232085] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10239.475643] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10246.254334] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10248.932469] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10257.622753] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10259.225848] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10265.065882] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10265.069846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10265.069969] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10265.079410] Lustre: Skipped 1 previous similar message [10265.100692] Lustre: Skipped 4 previous similar messages [10269.518991] Lustre: server umount lustre-OST0000 complete [10275.297345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10275.300569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10275.323634] Lustre: Skipped 1 previous similar message [10275.339732] Lustre: Skipped 1 previous similar message [10279.694806] Lustre: server umount lustre-MDT0000 complete [10280.420594] LustreError: 325256:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10280.435102] LustreError: 325256:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [10283.525875] LustreError: 321641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789413902 with bad export cookie 9371323216273537494 [10283.528884] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10283.546118] LustreError: 321641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10284.159911] Lustre: server umount lustre-MDT0001 complete [10297.381335] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [10300.784263] Key type lgssc unregistered [10301.065655] LNet: 327930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10301.070736] LNetError: 327930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10301.090623] LNet: Removed LNI 192.168.202.106@tcp [10301.956325] Key type .llcrypt unregistered [10301.958593] Key type ._llcrypt unregistered [10329.955279] Key type ._llcrypt registered [10329.958612] Key type .llcrypt registered [10330.062475] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [10330.922849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10330.939807] alg: No test for adler32 (adler32-zlib) [10331.897676] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [10332.101566] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [10333.751649] Key type lgssc registered [10335.254249] Lustre: Echo OBD driver; http://www.lustre.org/ [10343.829560] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10343.856974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10345.420939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10351.572438] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10355.685757] LustreError: 329345:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10355.698628] LustreError: 329345:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10360.800828] LustreError: 329344:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10361.266626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10361.753924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10366.551371] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10376.225860] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10383.114132] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10387.435518] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10387.438467] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10387.470438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10392.034381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10392.039560] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10393.023922] Lustre: server umount lustre-MDT0000 complete [10397.153849] LustreError: 330073:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10397.192993] LustreError: 330073:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10397.705698] LustreError: 329326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789414016 with bad export cookie 6716100748220079006 [10397.713412] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10397.744146] LustreError: 329326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10398.265750] Lustre: server umount lustre-MDT0001 complete [10416.352170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10416.661723] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10416.702733] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10416.906718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10422.254803] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10432.609764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10432.789784] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10432.856179] Lustre: 331936:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10433.236666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10439.774866] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10448.515508] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10453.400989] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10464.694610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10465.115588] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10465.123039] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10465.132153] LustreError: 333820:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [10465.138838] LustreError: 333820:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [10465.144415] LustreError: 333820:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10465.207780] Lustre: server umount lustre-OST0000 complete [10465.210977] LustreError: 333820:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10477.704771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10478.010338] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10478.298036] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10485.469852] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10486.062273] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10495.076143] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10503.551312] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10503.665367] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10503.801377] Lustre: lustre-OST0001: new disk, initializing [10503.811952] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10503.818625] Lustre: 335362:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10503.974682] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10509.970899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10509.980655] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10510.156601] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10514.356488] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10524.634821] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10530.802859] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10530.803641] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10530.854932] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10530.878508] Lustre: Skipped 1 previous similar message [10533.868313] Lustre: server umount lustre-OST0001 complete [10535.924681] LustreError: 335587:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10535.943727] LustreError: 335587:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10541.025492] LustreError: 335587:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10541.026911] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10541.048674] LustreError: 335587:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10541.065543] Lustre: Skipped 1 previous similar message [10541.071608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10541.081046] Lustre: Skipped 1 previous similar message [10546.157769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10551.267014] LustreError: 334288:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10551.288069] LustreError: 334288:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10552.287137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10552.520541] Lustre: server umount lustre-OST0000 complete [10561.520818] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10561.538208] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10561.548748] Lustre: Skipped 1 previous similar message [10561.549622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10561.549627] Lustre: Skipped 3 previous similar messages [10561.584644] LustreError: Skipped 1 previous similar message [10571.231154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10571.502845] Lustre: server umount lustre-MDT0000 complete [10571.748065] LustreError: 334282:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10571.811805] LustreError: 334282:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10577.526443] LustreError: 331927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789414196 with bad export cookie 6716100748220079552 [10577.541314] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10577.543339] LustreError: 331927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10578.178657] Lustre: server umount lustre-MDT0001 complete [10592.881515] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [10596.724135] Key type lgssc unregistered [10597.156963] LNet: 337325:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10597.165992] LNetError: 337325:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10597.193277] LNet: Removed LNI 192.168.202.106@tcp [10598.380313] Key type .llcrypt unregistered [10598.386281] Key type ._llcrypt unregistered [10680.068473] Key type ._llcrypt registered [10680.072635] Key type .llcrypt registered [10680.206293] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [10698.617814] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [10700.086532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10700.285469] alg: No test for adler32 (adler32-zlib) [10701.484958] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [10701.958491] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [10703.703935] Key type lgssc registered [10705.306468] Lustre: Echo OBD driver; http://www.lustre.org/ [10752.455029] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10752.487618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10753.914545] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10753.989565] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10754.220906] Lustre: lustre-MDT0000: new disk, initializing [10754.427054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10754.462864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10759.641679] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10773.018397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10773.175588] Lustre: 342259:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10773.254960] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10773.257876] Lustre: Skipped 1 previous similar message [10773.387311] Lustre: lustre-MDT0001: new disk, initializing [10773.548515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10773.600089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10773.616438] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10779.243662] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10791.323107] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10798.056056] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10806.613618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10806.951673] Lustre: lustre-OST0000: new disk, initializing [10806.957781] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10806.966268] Lustre: 344229:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10807.074284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10813.672914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10813.687745] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10813.858753] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10815.036542] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10826.320993] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10832.590448] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10832.950167] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10838.451964] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10838.659940] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10844.640422] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10844.643306] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10844.657950] LustreError: Skipped 1 previous similar message [10844.658729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10844.683657] Lustre: Skipped 1 previous similar message [10844.717595] Lustre: Skipped 1 previous similar message [10849.761332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10849.776870] Lustre: Skipped 1 previous similar message [10854.879183] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10854.887400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10854.911977] Lustre: Skipped 1 previous similar message [10855.216823] Lustre: server umount lustre-OST0000 complete [10860.007193] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10860.038130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10865.120463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10865.122843] Lustre: Skipped 1 previous similar message [10874.337407] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10874.754235] Lustre: server umount lustre-MDT0000 complete [10879.019607] LustreError: 342252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789414498 with bad export cookie 7110532144015568131 [10879.033369] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10879.041512] LustreError: 342252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10879.616484] Lustre: server umount lustre-MDT0001 complete [10890.163677] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 15:35:07 (1789414507) [10898.969373] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 15:35:16 (1789414516) [10909.173279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10909.928322] LustreError: 346989:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10909.953045] LustreError: 346989:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10910.075278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10915.309981] LustreError: 346990:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10915.334112] LustreError: 346990:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10916.851515] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10920.423921] LustreError: 346989:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10925.546375] LustreError: 346990:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10927.960714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10928.512863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10934.859569] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10944.261382] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10951.467590] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10961.240340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10961.959451] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10970.121326] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10980.274157] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10987.489134] LustreError: 346984:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2207613190210 [10987.501248] LustreError: 346984:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2207613190210 [10989.628585] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10992.021050] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10993.995959] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10996.085528] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10998.412402] Lustre: Failing over lustre-MDT0000 [10998.762351] Lustre: server umount lustre-MDT0000 complete [11000.291922] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11000.301916] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11000.307936] Lustre: Skipped 1 previous similar message [11000.313820] LustreError: 346990:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11008.598929] LustreError: 348521:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11008.632885] LustreError: 348521:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [11017.179928] Lustre: 340479:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789414620/real 1789414620] req@000000004d965f98 x1876336911137408/t0(0) o400->MGC192.168.202.106@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789414636 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11017.203989] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11020.179919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11027.425581] LustreError: 346990:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11027.469501] LustreError: 346990:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 17 previous similar messages [11028.449868] LustreError: 340476:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000c7f378d x1876336911143424/t0(0) o250->MGC192.168.202.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11028.739754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11028.797596] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11029.084236] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11033.276533] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11034.114532] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11034.175190] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [11034.235666] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11046.832281] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11049.674466] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11055.398547] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11058.686702] Lustre: Failing over lustre-MDT0000 [11059.053238] Lustre: server umount lustre-MDT0000 complete [11059.682764] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11059.687338] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11059.694160] LustreError: 347916:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11059.694171] LustreError: 347916:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11059.753395] Lustre: Skipped 3 previous similar messages [11075.043078] Lustre: 340479:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789414678/real 1789414678] req@0000000083ec86e0 x1876336911160576/t0(0) o400->MGC192.168.202.106@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789414694 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11075.074188] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11080.762535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11085.306370] LustreError: 340476:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f19872bd x1876336911165952/t0(0) o250->MGC192.168.202.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11085.714224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11085.821650] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11086.408755] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11090.946534] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11090.957104] Lustre: Skipped 2 previous similar messages [11090.980170] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [11091.031424] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11092.219908] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11105.222484] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11107.132928] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11116.524177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11116.524695] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11116.532046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11116.532053] Lustre: Skipped 2 previous similar messages [11116.536825] Lustre: Skipped 3 previous similar messages [11120.479889] Lustre: server umount lustre-MDT0000 complete [11124.669117] LustreError: 346970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789414743 with bad export cookie 7110532144015570840 [11124.676117] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11124.680106] LustreError: 346970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11125.235641] Lustre: server umount lustre-MDT0001 complete [11140.981098] Lustre: server umount lustre-OST0000 complete [11158.769746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11159.238825] LustreError: 353336:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11159.264323] LustreError: 353336:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 26 previous similar messages [11159.399401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11164.463730] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11174.300563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11180.676618] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11189.387756] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11196.735338] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11204.892155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11211.621547] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11220.553442] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11228.997801] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11239.392048] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11239.394899] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11239.419768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11239.426929] Lustre: Skipped 2 previous similar messages [11246.021540] Lustre: server umount lustre-OST0000 complete [11251.706435] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11257.096587] Lustre: server umount lustre-MDT0000 complete [11261.116342] LustreError: 353316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789414880 with bad export cookie 7110532144015571589 [11261.126697] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11261.129155] LustreError: 353316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11261.634637] Lustre: server umount lustre-MDT0001 complete [11274.112616] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [11277.644428] Key type lgssc unregistered [11278.034658] LNet: 356891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11278.038230] LNetError: 356891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11278.055780] LNet: Removed LNI 192.168.202.106@tcp [11279.285065] Key type .llcrypt unregistered [11279.290316] Key type ._llcrypt unregistered [11310.555314] Key type ._llcrypt registered [11310.559261] Key type .llcrypt registered [11310.764228] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [11312.051497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11312.233422] alg: No test for adler32 (adler32-zlib) [11313.254919] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [11313.512540] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [11315.248854] Key type lgssc registered [11316.755846] Lustre: Echo OBD driver; http://www.lustre.org/ [11326.012350] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11326.049161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11327.814773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11333.884557] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11338.216021] LustreError: 358409:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11338.228398] LustreError: 358409:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11343.000443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11343.329591] LustreError: 358410:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11343.693230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11347.898694] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11360.678783] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11367.730690] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11377.936353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11378.513441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11386.115393] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11395.805514] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11397.988448] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11407.644104] LustreError: 358404:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2207613190210 [11407.647694] LustreError: 358404:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2207613190210 [11415.528138] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11415.534945] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11415.555830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11418.593928] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11418.622551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11418.645789] Lustre: Skipped 2 previous similar messages [11420.781683] Lustre: server umount lustre-MDT0000 complete [11423.720626] LustreError: 358410:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11423.753261] LustreError: 358410:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11424.826404] LustreError: 358390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789415044 with bad export cookie 5326764600111985993 [11424.846636] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11424.853045] LustreError: 358390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11425.290656] Lustre: server umount lustre-MDT0001 complete [11440.563634] Lustre: server umount lustre-OST0000 complete [11455.882301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11456.651040] LustreError: 362150:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11456.780116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11462.113419] LustreError: 362151:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11462.797545] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11471.336906] LustreError: 362150:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11471.380475] LustreError: 362150:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11472.547735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11473.170646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11479.028804] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11487.125519] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11493.293497] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11501.251847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11501.747535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11509.034823] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11518.580721] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11526.280862] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11545.057983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11545.076146] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11545.109243] Lustre: Skipped 1 previous similar message [11545.133511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11547.117556] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11548.937411] Lustre: server umount lustre-MDT0000 complete [11552.231444] LustreError: 362151:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11552.255855] LustreError: 362151:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11553.271829] LustreError: 362131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789415172 with bad export cookie 5326764600111987106 [11553.290910] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11553.295203] LustreError: 362131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11554.221689] Lustre: server umount lustre-MDT0001 complete [11570.744566] Lustre: server umount lustre-OST0000 complete [11588.984386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11589.800618] LustreError: 366085:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11589.894886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11594.831798] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11605.319294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11611.946566] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11623.687684] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11631.495747] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11641.086085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11641.418990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11641.428478] Lustre: Skipped 1 previous similar message [11648.653159] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11659.394242] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11659.901658] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11666.405707] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11666.419183] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11666.437687] Lustre: Skipped 1 previous similar message [11666.440869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11666.461372] Lustre: Skipped 3 previous similar messages [11668.270498] Lustre: server umount lustre-OST0000 complete [11672.546333] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11672.557671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11675.628041] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11675.651689] Lustre: Skipped 1 previous similar message [11680.741894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11680.765152] Lustre: Skipped 3 previous similar messages [11681.949361] Lustre: server umount lustre-MDT0000 complete [11684.848482] LustreError: 366085:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11684.872213] LustreError: 366085:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [11686.652753] LustreError: 366066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789415305 with bad export cookie 5326764600111988128 [11686.666508] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11686.667033] LustreError: 366066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11687.169739] Lustre: server umount lustre-MDT0001 complete [11700.614355] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [11704.063939] Key type lgssc unregistered [11704.451174] LNet: 369618:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11704.459305] LNetError: 369618:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11704.485634] LNet: Removed LNI 192.168.202.106@tcp [11705.522737] Key type .llcrypt unregistered [11705.524050] Key type ._llcrypt unregistered [11737.930171] Key type ._llcrypt registered [11737.933413] Key type .llcrypt registered [11738.150176] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [11740.096996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11740.266869] alg: No test for adler32 (adler32-zlib) [11741.553291] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [11741.852310] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [11743.735396] Key type lgssc registered [11745.649678] Lustre: Echo OBD driver; http://www.lustre.org/ [11757.210143] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11757.246176] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11757.276387] LustreError: 371060:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11757.309975] LustreError: 371060:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11757.320404] LustreError: 371060:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11757.332576] LustreError: 371060:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11757.348370] LustreError: 371060:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11766.662117] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11766.672956] LustreError: 371457:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11766.709397] LustreError: 371457:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11766.720207] LustreError: 371457:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11766.728652] LustreError: 371457:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11766.749199] LustreError: 371457:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11790.749309] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [11794.226747] Key type lgssc unregistered [11794.566397] LNet: 372460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11794.570709] LNetError: 372460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11794.610864] LNet: Removed LNI 192.168.202.106@tcp [11795.699951] Key type .llcrypt unregistered [11795.702980] Key type ._llcrypt unregistered [11818.326424] Key type ._llcrypt registered [11818.336576] Key type .llcrypt registered [11818.479642] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [11838.238148] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [11839.638414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11839.734335] alg: No test for adler32 (adler32-zlib) [11840.806758] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [11841.062487] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [11842.775225] Key type lgssc registered [11844.386653] Lustre: Echo OBD driver; http://www.lustre.org/ [11888.495168] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11888.521497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11889.894266] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11889.944719] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11890.048119] Lustre: lustre-MDT0000: new disk, initializing [11890.161839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11890.216316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11896.165332] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11908.693077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11908.933380] Lustre: 375618:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11909.033831] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11909.037443] Lustre: Skipped 1 previous similar message [11909.097674] Lustre: lustre-MDT0001: new disk, initializing [11909.266241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11909.330667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11909.359276] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11915.158864] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11927.120907] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11934.088547] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11943.377304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11943.725646] Lustre: lustre-OST0000: new disk, initializing [11943.730760] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11943.737589] Lustre: 377589:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11943.853862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11952.650314] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11953.776465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11953.783276] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11954.013444] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11964.045542] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11970.509441] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11970.715813] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11976.809760] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11977.143240] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11979.744613] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11979.749742] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11979.778423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11979.784550] Lustre: Skipped 1 previous similar message [11984.880342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11984.890525] Lustre: Skipped 1 previous similar message [11985.909748] Lustre: server umount lustre-OST0000 complete [11991.021243] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11991.030449] LustreError: Skipped 1 previous similar message [11991.040378] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11991.073792] Lustre: Skipped 1 previous similar message [11991.090836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11995.115614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11995.118418] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11996.908769] Lustre: server umount lustre-MDT0000 complete [12001.773160] LustreError: 377588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789415621 with bad export cookie 6424070384164476998 [12001.784351] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12001.791588] LustreError: 377588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12002.434689] Lustre: server umount lustre-MDT0001 complete [12014.024538] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [12016.894193] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 15:53:53 (1789415633) [12025.960479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12026.805639] LustreError: 380048:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12026.832291] LustreError: 380048:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12026.961976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12031.970803] LustreError: 380049:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12033.303963] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12037.091142] LustreError: 380048:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12040.903841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12041.173155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12047.975689] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12058.083789] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12065.793843] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12076.006915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12076.465745] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12083.265390] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12092.409457] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12100.163670] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12100.520624] Lustre: lustre-OST0001: new disk, initializing [12100.523739] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12100.529123] Lustre: 382889:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12100.650541] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12106.469724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12106.486074] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12106.669612] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12109.231430] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12119.982255] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12127.231284] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12127.232052] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12127.235591] LustreError: Skipped 1 previous similar message [12127.239384] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12127.253726] Lustre: Skipped 1 previous similar message [12127.275673] Lustre: Skipped 2 previous similar messages [12130.868107] Lustre: server umount lustre-OST0001 complete [12132.339949] LustreError: 381875:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12132.374071] LustreError: 381875:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12137.446377] LustreError: 383146:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12137.500926] LustreError: 383146:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12147.707202] LustreError: 381876:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12147.769618] LustreError: 381876:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12152.806583] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12152.847484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12152.862987] Lustre: Skipped 2 previous similar messages [12156.826849] Lustre: server umount lustre-OST0000 complete [12162.167539] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12162.194679] Lustre: Skipped 1 previous similar message [12168.575958] Lustre: server umount lustre-MDT0000 complete [12172.770180] LustreError: 380048:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12172.786504] LustreError: 380048:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12173.830365] LustreError: 382888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789415793 with bad export cookie 6424070384164477887 [12173.849671] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12173.861916] LustreError: 382888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12180.412374] Lustre: server umount lustre-MDT0001 complete [12192.756413] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [12197.398427] Key type lgssc unregistered [12198.014968] LNet: 384878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12198.020928] LNetError: 384878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12199.079323] LNet: Removed LNI 192.168.202.106@tcp [12200.760290] Key type .llcrypt unregistered [12200.762315] Key type ._llcrypt unregistered [12264.551828] Key type ._llcrypt registered [12264.560121] Key type .llcrypt registered [12264.778221] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [12267.173230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12267.219660] alg: No test for adler32 (adler32-zlib) [12268.331526] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [12268.694563] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [12270.519351] Key type lgssc registered [12272.142505] Lustre: Echo OBD driver; http://www.lustre.org/ [12282.101230] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12282.132617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12283.415278] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12283.428124] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12283.532577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12289.113416] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12300.357055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12300.530721] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12300.560422] Lustre: 387679:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12300.754283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12306.707918] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12317.968443] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12327.277109] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12330.995445] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12331.062095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12331.066459] Lustre: Skipped 1 previous similar message [12336.102580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12336.112827] Lustre: Skipped 1 previous similar message [12337.769368] Lustre: server umount lustre-MDT0000 complete [12343.809664] LustreError: 387669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789415963 with bad export cookie 1878590724577400060 [12343.824256] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12343.828109] LustreError: 387669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12344.452929] Lustre: server umount lustre-MDT0001 complete [12356.832529] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 15:59:34 (1789415974) [12365.198790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12365.694423] LustreError: 390195:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12365.707826] LustreError: 390195:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12365.757455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12370.927320] LustreError: 390196:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12371.928091] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12376.039992] LustreError: 390195:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12381.159276] LustreError: 390196:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12381.648732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12382.267320] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12388.576407] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12400.091303] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12409.590468] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12419.811620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12420.101854] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12420.507461] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12429.021959] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12439.574765] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12450.074895] Lustre: Permanently deactivating lustre-OST0000 [12450.093790] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12458.111091] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12459.424672] LustreError: 393146:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12459.438823] LustreError: 393146:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12459.448238] LustreError: 393146:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12465.314468] LustreError: 393195:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12465.329088] LustreError: 393195:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12474.252900] Lustre: server umount lustre-OST0000 complete [12478.064397] LustreError: 393445:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12478.080371] LustreError: 393445:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12483.556940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12483.587114] Lustre: Skipped 1 previous similar message [12483.596723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12483.614267] Lustre: Skipped 1 previous similar message [12488.673345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12488.681373] Lustre: Skipped 1 previous similar message [12492.887905] Lustre: server umount lustre-MDT0000 complete [12493.800475] LustreError: 390196:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12493.818649] LustreError: 390196:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12498.243778] LustreError: 390176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789416117 with bad export cookie 1878590724577400641 [12498.246547] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12498.264809] LustreError: 390176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12508.852297] Lustre: server umount lustre-MDT0001 complete [12519.424680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12526.341761] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12529.655542] LustreError: 394607:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12537.450602] Lustre: 394947:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.106@tcp [12540.135329] Lustre: server umount MGS complete [12555.943366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12556.728299] LustreError: 395672:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12556.758355] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12556.765838] Lustre: Skipped 1 previous similar message [12556.859652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12563.779931] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12574.117194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12574.859026] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12575.007111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12580.620576] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12589.441837] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12595.727498] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12597.789648] Lustre: Permanently reactivating lustre-OST0000 [12597.802490] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12597.807303] Lustre: Skipped 2 previous similar messages [12623.839881] LustreError: 397404:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12636.526531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12637.330695] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12645.531812] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12654.463843] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12655.637598] LustreError: 395667:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255746 [12655.645824] LustreError: 395667:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255746 [12663.442318] Lustre: server umount lustre-OST0000 complete [12670.435533] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12670.437529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12670.459525] Lustre: Skipped 2 previous similar messages [12677.877740] Lustre: server umount lustre-MDT0000 complete [12680.680377] LustreError: 395672:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12680.726948] LustreError: 395672:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12680.803167] Lustre: 386999:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789416284/real 1789416284] req@0000000087c4adcd x1876338553235584/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789416300 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12680.857526] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12683.166524] LustreError: 395652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789416302 with bad export cookie 1878590724577401684 [12683.177154] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12683.186359] LustreError: 395652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12683.637751] Lustre: server umount lustre-MDT0001 complete [12696.740447] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [12699.157385] Key type lgssc unregistered [12699.425697] LNet: 399942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12699.432523] LNetError: 399942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12699.450498] LNet: Removed LNI 192.168.202.106@tcp [12700.211818] Key type .llcrypt unregistered [12700.213652] Key type ._llcrypt unregistered [12723.522856] Key type ._llcrypt registered [12723.525300] Key type .llcrypt registered [12723.708699] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [12738.942676] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [12740.453638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12740.565490] alg: No test for adler32 (adler32-zlib) [12741.853950] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [12742.104767] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [12743.783299] Key type lgssc registered [12745.873144] Lustre: Echo OBD driver; http://www.lustre.org/ [12792.114915] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12792.195142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12793.573597] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12793.603786] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12793.675853] Lustre: lustre-MDT0000: new disk, initializing [12793.741822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12793.761438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12800.113112] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12813.035725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12813.182204] Lustre: 403100:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12813.245576] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12813.251560] Lustre: Skipped 1 previous similar message [12813.357357] Lustre: lustre-MDT0001: new disk, initializing [12813.493120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12813.532844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12813.554669] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12818.619461] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12829.654698] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12835.775479] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12843.000599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12843.283979] Lustre: lustre-OST0000: new disk, initializing [12843.288386] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12843.301940] Lustre: 405070:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12843.485908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12850.930985] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12853.314531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12853.331150] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12853.562904] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12862.800899] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12869.829399] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12870.245649] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12875.109578] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12875.391838] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12879.332762] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12879.332964] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12879.335034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12879.339554] Lustre: Skipped 1 previous similar message [12879.344114] LustreError: Skipped 1 previous similar message [12883.640930] Lustre: server umount lustre-OST0000 complete [12889.575624] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12889.581917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12889.591203] Lustre: Skipped 1 previous similar message [12889.610140] Lustre: Skipped 2 previous similar messages [12894.906670] Lustre: server umount lustre-MDT0000 complete [12898.875971] LustreError: 403092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789416518 with bad export cookie 531767484292261384 [12898.877611] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12898.891822] LustreError: 403092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12899.275809] Lustre: server umount lustre-MDT0001 complete [12909.577419] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12911.535458] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 16:08:49 (1789416529) [12920.132311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12921.294522] LustreError: 407529:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12921.313845] LustreError: 407529:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [12921.442483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12926.432468] LustreError: 407530:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12928.684468] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12931.558773] LustreError: 407529:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12936.675107] LustreError: 407530:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12940.929134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12941.732524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12946.483127] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12954.840729] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12961.300260] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12969.455755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12969.741333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12979.460105] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12988.291203] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12990.808989] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12994.977039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12995.011985] LustreError: 409973:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2207613190210 [12995.016926] LustreError: 409973:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2207613190210 [13005.761849] Lustre: server umount lustre-OST0000 complete [13011.938219] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13011.954833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13011.975688] Lustre: Skipped 1 previous similar message [13012.006836] Lustre: Skipped 1 previous similar message [13017.060158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13017.069905] Lustre: Skipped 1 previous similar message [13022.232717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13022.238579] Lustre: Skipped 1 previous similar message [13023.264525] Lustre: 401320:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789416626/real 1789416626] req@00000000e3ea98d6 x1876339050225664/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789416642 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13023.288912] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13025.248023] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13025.467814] Lustre: server umount lustre-MDT0000 complete [13028.326425] LustreError: 407526:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13028.344513] LustreError: 407526:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13028.384896] Lustre: 401319:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789416631/real 1789416631] req@00000000fbe403f4 x1876339050227968/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789416647 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13029.823491] LustreError: 407511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789416649 with bad export cookie 531767484292262266 [13029.831832] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13029.831964] LustreError: 407511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13030.295623] Lustre: server umount lustre-MDT0001 complete [13045.422273] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [13050.662907] Key type lgssc unregistered [13051.411892] LNet: 411258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13051.422591] LNetError: 411258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13051.453839] LNet: Removed LNI 192.168.202.106@tcp [13052.771246] Key type .llcrypt unregistered [13052.772902] Key type ._llcrypt unregistered [13087.936256] Key type ._llcrypt registered [13087.943654] Key type .llcrypt registered [13088.167486] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [13088.587394] Key type .llcrypt unregistered [13088.589041] Key type ._llcrypt unregistered [13111.387852] Key type ._llcrypt registered [13111.391971] Key type .llcrypt registered [13111.551444] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [13113.203339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13113.612706] alg: No test for adler32 (adler32-zlib) [13114.703847] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [13115.056800] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [13116.847313] Key type lgssc registered [13118.708624] Lustre: Echo OBD driver; http://www.lustre.org/ [13130.250397] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13130.293031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13131.880148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13136.974669] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [13142.500895] LustreError: 413468:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13142.519203] LustreError: 413468:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13145.489431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13145.738951] LustreError: 413469:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13145.770027] LustreError: 413469:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13145.776409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13151.215939] LustreError: 413468:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13151.234641] LustreError: 413468:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13152.061159] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [13156.323160] LustreError: 413464:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13156.339883] LustreError: 413464:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13157.367991] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13160.140501] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13162.468898] LustreError: 413469:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13162.492583] LustreError: 413469:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [13167.512693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13168.103509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13173.938234] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [13182.946750] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13182.951895] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13182.975985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13183.478215] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13183.494379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13183.511082] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13183.512843] Lustre: Skipped 1 previous similar message [13184.982236] Lustre: server umount lustre-OST0000 complete [13191.138696] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13191.141315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13191.146789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13193.699416] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13193.705152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13193.717305] Lustre: Skipped 1 previous similar message [13195.288588] Lustre: server umount lustre-MDT0000 complete [13198.817261] LustreError: 413468:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13198.851650] LustreError: 413468:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [13199.009394] LustreError: 413448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789416818 with bad export cookie 18194814118278565971 [13199.020289] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13199.021601] LustreError: 413448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13199.762697] Lustre: server umount lustre-MDT0001 complete [13217.472634] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [13221.317869] Key type lgssc unregistered [13222.000762] LNet: 416618:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13222.026174] LNetError: 416618:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13222.050666] LNet: Removed LNI 192.168.202.106@tcp [13223.388249] Key type .llcrypt unregistered [13223.395134] Key type ._llcrypt unregistered [13253.878684] Key type ._llcrypt registered [13253.883448] Key type .llcrypt registered [13254.010490] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [13255.084580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13255.124055] alg: No test for adler32 (adler32-zlib) [13256.468803] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [13256.835037] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [13258.577505] Key type lgssc registered [13260.231913] Lustre: Echo OBD driver; http://www.lustre.org/ [13271.163523] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13271.177767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13286.688171] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13291.807509] LustreError: 418007:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.106@tcp: failed processing log, type 4: rc = -110 [13318.500260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13325.005856] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [13332.589918] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13338.600193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13339.123811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13339.170324] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13344.237377] LustreError: 418033:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13344.275270] LustreError: 418033:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13346.124216] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [13349.358172] LustreError: 418032:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13353.521162] LustreError: 419045:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13353.553833] LustreError: 419045:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13356.678931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13357.282683] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13361.482963] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [13370.132679] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13376.037157] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13385.186865] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13385.200956] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13385.219025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13387.745166] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13387.755906] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13387.769859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13393.378820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13393.403140] Lustre: Skipped 3 previous similar messages [13397.985277] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13398.284338] Lustre: server umount lustre-OST0000 complete [13403.616622] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13403.640929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13408.225293] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13408.238442] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13408.253904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13409.912436] Lustre: server umount lustre-MDT0000 complete [13413.856920] LustreError: 419044:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13413.881110] LustreError: 419044:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13413.912673] LustreError: 418013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789417033 with bad export cookie 3747656508228038231 [13413.935654] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13414.415624] Lustre: server umount lustre-MDT0001 complete [13425.656655] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [13428.441660] Key type lgssc unregistered [13428.788632] LNet: 421580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13428.792453] LNetError: 421580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13429.886893] LNet: Removed LNI 192.168.202.106@tcp [13431.085144] Key type .llcrypt unregistered [13431.088792] Key type ._llcrypt unregistered [13456.148513] Key type ._llcrypt registered [13456.150552] Key type .llcrypt registered [13456.244879] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [13457.414152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13457.569068] alg: No test for adler32 (adler32-zlib) [13458.862382] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [13459.308610] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [13461.071208] Key type lgssc registered [13461.925737] Lustre: Echo OBD driver; http://www.lustre.org/ [13468.532359] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13468.561531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13470.129891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13474.275150] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [13480.424816] LustreError: 422995:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13480.447910] LustreError: 422995:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13483.759706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13484.257103] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13492.273170] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [13500.399559] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13506.283496] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13514.310111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13514.747682] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13515.796234] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13521.421522] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [13530.696716] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13538.932726] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.6@tcp (stopping) [13539.298900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13539.307874] LustreError: 422992:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13539.320393] LustreError: 422992:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13539.362609] Lustre: server umount lustre-MDT0000 complete [13542.720770] LustreError: 422974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.6@tcp arrived at 1789417162 with bad export cookie 3466479969641450150 [13542.750700] LustreError: 422974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13544.418749] LustreError: 422995:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13544.460814] LustreError: 422995:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13545.664085] LustreError: 422975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789417164 with bad export cookie 3466479969641449254 [13545.689039] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13545.692053] LustreError: 422975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13549.537133] LustreError: 424072:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13549.539752] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13549.571683] LustreError: 424072:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13549.599657] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13549.620170] Lustre: Skipped 2 previous similar messages [13549.633649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13551.978850] Lustre: server umount lustre-OST0000 complete [13555.701552] LustreError: 422995:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13555.713327] LustreError: 422995:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [13559.157403] Lustre: server umount lustre-MDT0001 complete [13572.047397] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [13575.506914] Key type lgssc unregistered [13575.781171] LNet: 426679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13575.803401] LNetError: 426679:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13576.870940] LNet: Removed LNI 192.168.202.106@tcp [13577.878348] Key type .llcrypt unregistered [13577.881404] Key type ._llcrypt unregistered [13611.799547] Key type ._llcrypt registered [13611.801086] Key type .llcrypt registered [13612.020654] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [13613.763444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13613.861721] alg: No test for adler32 (adler32-zlib) [13615.470494] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [13615.893108] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [13617.663236] Key type lgssc registered [13618.807956] Lustre: Echo OBD driver; http://www.lustre.org/ [13627.982579] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13628.004379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13629.492411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13634.886192] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [13639.658858] LustreError: 428095:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13639.701746] LustreError: 428095:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13644.802265] LustreError: 428094:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13648.501145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13649.024824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13655.475240] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [13666.151548] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13673.010126] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13681.242835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13681.625383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13688.986251] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [13698.167844] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13699.123709] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13704.166674] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13704.181880] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13704.191432] Lustre: Skipped 1 previous similar message [13706.331756] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.6@tcp (stopping) [13709.286655] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13709.293944] Lustre: Skipped 1 previous similar message [13709.883979] Lustre: server umount lustre-MDT0001 complete [13711.436164] LustreError: 428089:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13714.411422] LustreError: 428823:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13714.430304] LustreError: 428823:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13719.521196] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13719.524626] LustreError: 428095:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13719.530685] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13719.530693] Lustre: Skipped 1 previous similar message [13719.534783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13722.730920] Lustre: server umount lustre-OST0000 complete [13729.936721] Lustre: server umount lustre-MDT0000 complete [13750.963665] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [13755.320453] Key type lgssc unregistered [13755.766724] LNet: 431782:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13755.774460] LNetError: 431782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13755.791135] LNet: Removed LNI 192.168.202.106@tcp [13756.692512] Key type .llcrypt unregistered [13756.694416] Key type ._llcrypt unregistered [13788.770954] Key type ._llcrypt registered [13788.772628] Key type .llcrypt registered [13788.829131] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [13805.954645] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [13808.200062] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13808.305142] alg: No test for adler32 (adler32-zlib) [13809.530562] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [13809.850164] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [13811.600118] Key type lgssc registered [13813.420802] Lustre: Echo OBD driver; http://www.lustre.org/ [13858.377916] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [13875.029253] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13875.057852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13876.397121] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13876.432140] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13876.580786] Lustre: lustre-MDT0000: new disk, initializing [13876.736538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13876.772416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13882.459458] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [13904.459408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13905.129979] Lustre: lustre-OST0000: new disk, initializing [13905.146770] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13905.155902] Lustre: Skipped 1 previous similar message [13905.169417] Lustre: 436819:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13905.333684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13912.206144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13912.215804] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13912.238513] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13914.149663] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [13925.135276] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13945.390487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13945.565608] Lustre: 435799:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13945.676342] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13945.796295] Lustre: lustre-MDT0001: new disk, initializing [13945.985111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13946.038356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13946.061254] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13951.360845] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [13962.039581] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13962.293694] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13966.562461] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13966.815701] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13976.193082] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.6@tcp (stopping) [13976.548663] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13976.551820] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13978.691236] Lustre: server umount lustre-MDT0001 complete [13986.803041] LustreError: 435804:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13986.854982] LustreError: 435804:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [13991.906986] LustreError: 435805:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13991.919940] LustreError: 435805:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13997.030298] LustreError: 435809:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13997.031345] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13997.113551] Lustre: Skipped 1 previous similar message [13997.156133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13997.174486] Lustre: Skipped 2 previous similar messages [14002.145566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14002.146892] LustreError: 435810:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14002.839830] Lustre: server umount lustre-OST0000 complete [14007.275044] LustreError: 435809:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14016.263412] Lustre: server umount lustre-MDT0000 complete [14034.161209] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [14037.350288] Key type lgssc unregistered [14037.698388] LNet: 440568:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14037.706484] LNetError: 440568:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14037.736358] LNet: Removed LNI 192.168.202.106@tcp [14038.786565] Key type .llcrypt unregistered [14038.787581] Key type ._llcrypt unregistered [14086.682268] Key type ._llcrypt registered [14086.690363] Key type .llcrypt registered [14086.985585] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [14087.532434] Key type .llcrypt unregistered [14087.534915] Key type ._llcrypt unregistered [14110.188089] Key type ._llcrypt registered [14110.194045] Key type .llcrypt registered [14110.339665] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [14111.264494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14111.285142] alg: No test for adler32 (adler32-zlib) [14112.309850] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [14112.566928] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [14114.232758] Key type lgssc registered [14115.942178] Lustre: Echo OBD driver; http://www.lustre.org/ [14150.694702] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14150.741423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14152.218190] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14152.308279] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14152.508668] Lustre: lustre-MDT0000: new disk, initializing [14152.737832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14152.768063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14160.981797] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [14172.525239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14172.714615] Lustre: 443943:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14172.792539] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14172.796698] Lustre: Skipped 1 previous similar message [14172.912438] Lustre: lustre-MDT0001: new disk, initializing [14173.007526] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14173.070961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14173.089159] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14178.614981] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [14189.369979] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14197.196486] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14206.353773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14206.626317] Lustre: lustre-OST0000: new disk, initializing [14206.629796] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14206.636931] Lustre: 445911:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14206.743336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14212.825675] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [14214.244373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14214.258917] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14214.533751] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14224.296975] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14226.143371] LustreError: 443948:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255619 [14226.150065] LustreError: 443948:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255619 [14234.598147] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14234.602653] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14234.610318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14235.105253] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14235.119313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14235.122214] Lustre: Skipped 1 previous similar message [14236.928513] Lustre: server umount lustre-MDT0000 complete [14240.425883] LustreError: 443933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789417859 with bad export cookie 9183701103497686242 [14240.428643] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14240.442299] LustreError: 443933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14240.735250] Lustre: server umount lustre-MDT0001 complete [14255.544206] Lustre: server umount lustre-OST0000 complete [14267.719973] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 16:31:25 (1789417885) [14277.618084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14278.389126] LustreError: 447939:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14278.406131] LustreError: 447939:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [14278.460758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14283.758022] LustreError: 447940:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14283.966307] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [14287.909336] LustreError: 447939:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14292.974295] LustreError: 447940:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14293.148410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14293.701644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14299.332141] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [14307.396576] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14312.872388] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14332.980336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14333.184256] Lustre: Found index 0 for lustre-OST0000, updating log [14333.412519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14342.224258] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [14349.828966] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14350.193582] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14353.888379] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14353.901045] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14353.932070] Lustre: Skipped 1 previous similar message [14353.953967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14359.738257] Lustre: server umount lustre-OST0000 complete [14380.512460] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14380.520117] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14380.531555] Lustre: Skipped 1 previous similar message [14380.538140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14380.546987] Lustre: Skipped 3 previous similar messages [14383.648441] LustreError: 447940:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14383.678074] LustreError: 447940:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [14383.776114] Lustre: server umount lustre-MDT0000 complete [14387.193106] LustreError: 449785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789418006 with bad export cookie 9183701103497687635 [14387.198377] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14387.198841] LustreError: 449785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14387.489720] Lustre: server umount lustre-MDT0001 complete [14388.323581] LNet: 1 local NIs in recovery (showing 1): 192.168.202.106@tcp [14395.465363] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 16:33:33 (1789418013) [15023.803449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15024.328730] LustreError: 454350:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15024.400891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15029.593231] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [15029.736246] LustreError: 454351:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15033.833314] LustreError: 454350:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15038.038851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15038.365239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15043.952519] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [15052.133705] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15058.292628] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15079.088933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15079.462370] Lustre: Found index 0 for lustre-OST0000, updating log [15079.520529] LNet: 1 local NIs in recovery (showing 1): 192.168.202.106@tcp [15079.959452] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15087.181130] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [15096.402337] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15099.684581] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15099.692698] Lustre: Skipped 1 previous similar message [15099.697788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15099.703604] Lustre: Skipped 4 previous similar messages [15105.244951] Lustre: server umount lustre-OST0000 complete [15110.117421] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15110.125793] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15110.137450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15111.205323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15111.207092] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15113.253129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15113.259105] Lustre: Skipped 4 previous similar messages [15115.802772] Lustre: server umount lustre-MDT0000 complete [15116.322624] LustreError: 454350:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15116.329254] LustreError: 454350:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [15119.040937] LustreError: 454330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789418738 with bad export cookie 9183701103497688720 [15119.043595] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15119.045851] LustreError: 454330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15119.500618] Lustre: server umount lustre-MDT0001 complete [15138.911266] LNet: 1 local NIs in recovery (showing 1): 192.168.202.106@tcp [15150.738823] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [15165.505305] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [15210.269466] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 16:47:07 (1789418827) [15232.283231] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [15236.609772] Key type lgssc unregistered [15237.103749] LNet: 460971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15237.107262] LNetError: 460971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15340.585980] LNet: Removed LNI 192.168.202.106@tcp [15341.888181] Key type .llcrypt unregistered [15341.904201] Key type ._llcrypt unregistered [15365.772541] Key type ._llcrypt registered [15365.774650] Key type .llcrypt registered [15365.951166] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [15367.806783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15367.836469] alg: No test for adler32 (adler32-zlib) [15368.985348] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [15369.279254] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [15371.016217] Key type lgssc registered [15372.280993] Lustre: Echo OBD driver; http://www.lustre.org/ [15377.635978] LNet: Added LNI 192.168.202.106@tcp100 [8/256/0/180] [15378.642529] LNet: Added LNI 192.168.202.106@tcp101 [8/256/0/180] [15379.513474] LNet: Added LNI 192.168.202.106@tcp102 [8/256/0/180] [15380.594865] LNet: Added LNI 192.168.202.106@tcp103 [8/256/0/180] [15381.767434] LNet: Added LNI 192.168.202.106@tcp104 [8/256/0/180] [15382.770355] LNet: Added LNI 192.168.202.106@tcp105 [8/256/0/180] [15383.725933] LNet: Added LNI 192.168.202.106@tcp106 [8/256/0/180] [15384.705256] LNet: Added LNI 192.168.202.106@tcp107 [8/256/0/180] [15385.665052] LNet: Added LNI 192.168.202.106@tcp108 [8/256/0/180] [15386.629388] LNet: Added LNI 192.168.202.106@tcp109 [8/256/0/180] [15387.534620] LNet: Added LNI 192.168.202.106@tcp110 [8/256/0/180] [15388.550188] LNet: Added LNI 192.168.202.106@tcp111 [8/256/0/180] [15389.479685] LNet: Added LNI 192.168.202.106@tcp112 [8/256/0/180] [15390.310740] LNet: Added LNI 192.168.202.106@tcp113 [8/256/0/180] [15391.061415] LNet: Added LNI 192.168.202.106@tcp114 [8/256/0/180] [15391.973602] LNet: Added LNI 192.168.202.106@tcp115 [8/256/0/180] [15392.919501] LNet: Added LNI 192.168.202.106@tcp116 [8/256/0/180] [15393.818564] LNet: Added LNI 192.168.202.106@tcp117 [8/256/0/180] [15394.623186] LNet: Added LNI 192.168.202.106@tcp118 [8/256/0/180] [15395.361338] LNet: Added LNI 192.168.202.106@tcp119 [8/256/0/180] [15395.940965] LNet: Added LNI 192.168.202.106@tcp120 [8/256/0/180] [15396.630762] LNet: Added LNI 192.168.202.106@tcp121 [8/256/0/180] [15397.483788] LNet: Added LNI 192.168.202.106@tcp122 [8/256/0/180] [15398.480044] LNet: Added LNI 192.168.202.106@tcp123 [8/256/0/180] [15399.441050] LNet: Added LNI 192.168.202.106@tcp124 [8/256/0/180] [15400.505763] LNet: Added LNI 192.168.202.106@tcp125 [8/256/0/180] [15401.510698] LNet: Added LNI 192.168.202.106@tcp126 [8/256/0/180] [15402.235642] LNet: Added LNI 192.168.202.106@tcp127 [8/256/0/180] [15403.081340] LNet: Added LNI 192.168.202.106@tcp128 [8/256/0/180] [15403.837060] LNet: Added LNI 192.168.202.106@tcp129 [8/256/0/180] [15404.910454] LNet: Added LNI 192.168.202.106@tcp130 [8/256/0/180] [15405.840189] LNet: Added LNI 192.168.202.106@tcp131 [8/256/0/180] [15406.698403] LNet: Added LNI 192.168.202.106@tcp132 [8/256/0/180] [15407.574323] LNet: Added LNI 192.168.202.106@tcp133 [8/256/0/180] [15409.042934] LNet: Added LNI 192.168.202.106@tcp134 [8/256/0/180] [15410.073612] LNet: Added LNI 192.168.202.106@tcp135 [8/256/0/180] [15418.282331] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15418.365105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15419.590910] Lustre: 464003:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.106@tcp: can't add NID 192.168.202.106@tcp130: rc = -75 [15419.740974] LNetError: 461459:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.106 [15419.781257] LNetError: Protocol error connecting to 192.168.202.106@tcp118 at host 192.168.202.106:988: is it running a compatible version of Lustre? [15419.806598] LNetError: Skipped 3 previous similar messages [15419.917369] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15419.991556] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15420.147753] Lustre: lustre-MDT0000: new disk, initializing [15420.274549] Lustre: 464062:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.106@tcp: can't add NID 192.168.202.106@tcp130: rc = -75 [15420.297937] Lustre: 464062:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15420.325565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15420.357473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15420.519381] LNet: 4 local NIs in recovery (showing 4): 192.168.202.106@tcp126, 192.168.202.106@tcp127, 192.168.202.106@tcp129, 192.168.202.106@tcp128 [15420.523600] LNetError: 464034:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.106 [15420.574556] LNetError: 464034:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [15420.593567] LNetError: Protocol error connecting to 192.168.202.106@tcp100 at host 192.168.202.106:988: is it running a compatible version of Lustre? [15420.609413] LNetError: Skipped 9 previous similar messages [15421.600336] LNetError: 461461:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.106 [15421.607820] LNetError: 461461:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 3 previous similar messages [15421.617036] LNetError: Protocol error connecting to 192.168.202.106@tcp128 at host 192.168.202.106:988: is it running a compatible version of Lustre? [15426.720708] LNetError: 464025:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.106 [15426.732580] LNetError: 464025:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [15426.739327] LNetError: Protocol error connecting to 192.168.202.106@tcp127 at host 192.168.202.106:988: is it running a compatible version of Lustre? [15426.746881] LNetError: Skipped 1 previous similar message [15426.901151] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [15434.912620] LNetError: 464039:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.106 [15434.940382] LNetError: Protocol error connecting to 192.168.202.106@tcp127 at host 192.168.202.106:988: is it running a compatible version of Lustre? [15437.919323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15438.140944] Lustre: 464019:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15438.214511] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15438.217315] Lustre: Skipped 1 previous similar message [15438.326260] Lustre: lustre-MDT0001: new disk, initializing [15438.387339] Lustre: 464823:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.106@tcp: can't add NID 192.168.202.106@tcp130: rc = -75 [15438.403810] Lustre: 464823:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15438.578024] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15438.644261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15438.667094] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15444.187455] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [15446.643324] Lustre: 465187:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.106@tcp: can't add NID 192.168.202.106@tcp130: rc = -75 [15446.656095] Lustre: 465187:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15450.361594] LNetError: 464026:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.106 [15450.383902] LNetError: Protocol error connecting to 192.168.202.106@tcp127 at host 192.168.202.106:988: is it running a compatible version of Lustre? [15454.139635] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15460.764656] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15466.793115] LNetError: 461461:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.106 [15466.815314] LNetError: 461461:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 11 previous similar messages [15466.836209] LNetError: Protocol error connecting to 192.168.202.106@tcp119 at host 192.168.202.106:988: is it running a compatible version of Lustre? [15466.860262] LNetError: Skipped 11 previous similar messages [15468.494265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15468.705130] Lustre: lustre-OST0000: new disk, initializing [15468.708423] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15468.712651] Lustre: 465996:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15468.791292] Lustre: 466001:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.106@tcp: can't add NID 192.168.202.106@tcp130: rc = -75 [15468.796171] Lustre: 466001:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15468.835214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15477.243365] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [15477.798801] Lustre: 466370:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.106@tcp: can't add NID 192.168.202.106@tcp130: rc = -75 [15477.808333] Lustre: 466370:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15477.902929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15477.921335] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15478.079661] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15480.031096] LNet: 1 local NIs in recovery (showing 1): 192.168.202.106@tcp127 [15490.565886] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15499.616480] LNetError: 464039:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.106 [15499.630655] LNetError: 464039:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 18 previous similar messages [15499.642745] LNetError: Protocol error connecting to 192.168.202.106@tcp118 at host 192.168.202.106:988: is it running a compatible version of Lustre? [15499.657358] LNetError: Skipped 18 previous similar messages [15503.850395] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15503.851545] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15503.856224] LustreError: Skipped 1 previous similar message [15503.857060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15503.875189] Lustre: Skipped 1 previous similar message [15503.924732] Lustre: Skipped 1 previous similar message [15505.884181] Lustre: server umount lustre-OST0000 complete [15514.090391] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15514.093152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15514.117042] Lustre: Skipped 1 previous similar message [15514.131652] Lustre: Skipped 1 previous similar message [15516.980744] Lustre: server umount lustre-MDT0000 complete [15521.002370] LustreError: 464011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789419140 with bad export cookie 16743277813169694167 [15521.017401] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15521.523234] Lustre: server umount lustre-MDT0001 complete [15532.979821] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [15535.957265] Key type lgssc unregistered [15536.433221] LNet: 467912:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15536.461112] LNetError: 467912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15536.467950] LNet: Removed LNI 192.168.202.106@tcp [15537.506333] LNet: Removed LNI 192.168.202.106@tcp100 [15537.509321] LNetError: 467912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15537.522376] LNetError: 467912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15537.533151] LNet: Removed LNI 192.168.202.106@tcp101 [15538.591135] LNet: Removed LNI 192.168.202.106@tcp102 [15538.599693] LNetError: 467912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15538.622427] LNetError: 467912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15538.633090] LNet: Removed LNI 192.168.202.106@tcp103 [15539.679202] LNet: Removed LNI 192.168.202.106@tcp104 [15540.704189] LNet: Removed LNI 192.168.202.106@tcp105 [15540.713619] LNetError: 467912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15540.730707] LNetError: 467912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15540.756592] LNet: Removed LNI 192.168.202.106@tcp106 [15541.791207] LNet: Removed LNI 192.168.202.106@tcp107 [15542.815739] LNet: Removed LNI 192.168.202.106@tcp108 [15543.840709] LNet: Removed LNI 192.168.202.106@tcp109 [15544.863161] LNet: Removed LNI 192.168.202.106@tcp110 [15544.870648] LNetError: 467912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15544.880204] LNetError: 467912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15544.886808] LNet: Removed LNI 192.168.202.106@tcp111 [15545.951299] LNet: Removed LNI 192.168.202.106@tcp112 [15546.975174] LNet: Removed LNI 192.168.202.106@tcp113 [15548.002028] LNet: Removed LNI 192.168.202.106@tcp114 [15549.023173] LNet: Removed LNI 192.168.202.106@tcp115 [15550.047167] LNet: Removed LNI 192.168.202.106@tcp116 [15551.072520] LNet: Removed LNI 192.168.202.106@tcp117 [15552.095816] LNet: Removed LNI 192.168.202.106@tcp118 [15553.119800] LNet: Removed LNI 192.168.202.106@tcp119 [15553.139191] LNetError: 467912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15553.155041] LNetError: 467912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15553.163794] LNet: Removed LNI 192.168.202.106@tcp120 [15554.207894] LNet: Removed LNI 192.168.202.106@tcp121 [15555.231713] LNet: Removed LNI 192.168.202.106@tcp122 [15556.255230] LNet: Removed LNI 192.168.202.106@tcp123 [15557.282054] LNet: Removed LNI 192.168.202.106@tcp124 [15558.307124] LNet: Removed LNI 192.168.202.106@tcp125 [15559.391474] LNet: Removed LNI 192.168.202.106@tcp126 [15559.398685] LNet: Removed LNI 192.168.202.106@tcp127 [15560.415338] LNet: Removed LNI 192.168.202.106@tcp128 [15561.439235] LNet: Removed LNI 192.168.202.106@tcp129 [15561.446274] LNet: Removed LNI 192.168.202.106@tcp130 [15561.448780] LNet: Removed LNI 192.168.202.106@tcp131 [15561.453739] LNet: Removed LNI 192.168.202.106@tcp132 [15561.456992] LNet: Removed LNI 192.168.202.106@tcp133 [15561.461683] LNet: Removed LNI 192.168.202.106@tcp134 [15561.496840] LNet: Removed LNI 192.168.202.106@tcp135 [15562.653463] Key type .llcrypt unregistered [15562.655765] Key type ._llcrypt unregistered [15597.109289] Key type ._llcrypt registered [15597.111556] Key type .llcrypt registered [15597.267520] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [15615.848832] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [15617.436275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15617.475194] alg: No test for adler32 (adler32-zlib) [15618.572685] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [15618.848269] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [15620.535384] Key type lgssc registered [15622.146305] Lustre: Echo OBD driver; http://www.lustre.org/ [15666.021460] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 16:54:42 (1789419282) [15693.533401] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [15698.396906] Key type lgssc unregistered [15698.858156] LNet: 472017:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15698.866763] LNetError: 472017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15698.889875] LNet: Removed LNI 192.168.202.106@tcp [15700.118173] Key type .llcrypt unregistered [15700.120159] Key type ._llcrypt unregistered [15720.843564] Key type ._llcrypt registered [15720.845630] Key type .llcrypt registered [15720.968739] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [15722.103145] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15722.282463] alg: No test for adler32 (adler32-zlib) [15723.341953] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [15723.548741] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [15725.199829] Key type lgssc registered [15726.668264] Lustre: Echo OBD driver; http://www.lustre.org/ [15737.982570] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15738.104781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15739.493158] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15739.559389] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15739.663802] Lustre: lustre-MDT0000: new disk, initializing [15739.820930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15739.847917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15745.157938] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [15758.808085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15759.007093] Lustre: 473229:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15759.095223] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15759.103586] Lustre: Skipped 1 previous similar message [15759.234079] Lustre: lustre-MDT0001: new disk, initializing [15759.371246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15759.495371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15759.518594] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15764.824631] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [15775.326723] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15782.650347] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15792.864359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15793.330729] Lustre: lustre-OST0000: new disk, initializing [15793.335809] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15793.342550] Lustre: 475201:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15793.428344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15798.884846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15798.907957] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15799.051145] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15801.465902] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [15814.551099] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15820.284053] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing lnet_if_list [15827.345959] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15827.404087] LNet: Added LNI 192.168.202.106@tcp10 [8/256/0/180] [15835.293683] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15835.343051] LNet: Added LNI 192.168.202.106@tcp11 [8/256/0/180] [15841.332810] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15841.380090] LNet: Added LNI 192.168.202.106@tcp12 [8/256/0/180] [15846.912410] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15846.945434] LNet: Added LNI 192.168.202.106@tcp13 [8/256/0/180] [15852.345732] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15852.378707] LNet: Added LNI 192.168.202.106@tcp14 [8/256/0/180] [15858.536179] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15858.576833] LNet: Added LNI 192.168.202.106@tcp15 [8/256/0/180] [15865.018992] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15865.052194] LNet: Added LNI 192.168.202.106@tcp16 [8/256/0/180] [15870.356776] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15870.386795] LNet: Added LNI 192.168.202.106@tcp17 [8/256/0/180] [15876.147608] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15876.194825] LNet: Added LNI 192.168.202.106@tcp18 [8/256/0/180] [15882.769181] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15882.808153] LNet: Added LNI 192.168.202.106@tcp19 [8/256/0/180] [15890.327851] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15890.394265] LNet: Added LNI 192.168.202.106@tcp20 [8/256/0/180] [15896.712245] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15896.765563] LNet: Added LNI 192.168.202.106@tcp21 [8/256/0/180] [15904.018309] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15904.061578] LNet: Added LNI 192.168.202.106@tcp22 [8/256/0/180] [15912.884621] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15912.923253] LNet: Added LNI 192.168.202.106@tcp23 [8/256/0/180] [15920.417111] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15920.458513] LNet: Added LNI 192.168.202.106@tcp24 [8/256/0/180] [15929.543050] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15929.567520] LNet: Added LNI 192.168.202.106@tcp25 [8/256/0/180] [15937.337964] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15937.392259] LNet: Added LNI 192.168.202.106@tcp26 [8/256/0/180] [15945.904179] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15945.973653] LNet: Added LNI 192.168.202.106@tcp27 [8/256/0/180] [15954.611663] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15954.689570] LNet: Added LNI 192.168.202.106@tcp28 [8/256/0/180] [15964.869238] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15964.958350] LNet: Added LNI 192.168.202.106@tcp29 [8/256/0/180] [15974.803690] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15974.871621] LNet: Added LNI 192.168.202.106@tcp30 [8/256/0/180] [15983.393671] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15983.448502] LNet: Added LNI 192.168.202.106@tcp31 [8/256/0/180] [15992.420936] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15992.480711] LNet: Added LNI 192.168.202.106@tcp32 [8/256/0/180] [16000.731790] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [16000.777907] LNet: Added LNI 192.168.202.106@tcp33 [8/256/0/180] [16009.784325] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [16009.829965] LNet: Added LNI 192.168.202.106@tcp34 [8/256/0/180] [16017.211517] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [16017.231896] LNet: Added LNI 192.168.202.106@tcp35 [8/256/0/180] [16024.851380] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [16024.903540] LNet: Added LNI 192.168.202.106@tcp36 [8/256/0/180] [16037.284772] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [16037.332258] LNet: Added LNI 192.168.202.106@tcp37 [8/256/0/180] [16047.605913] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [16047.648394] LNet: Added LNI 192.168.202.106@tcp38 [8/256/0/180] [16057.056587] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [16057.104520] LNet: Added LNI 192.168.202.106@tcp39 [8/256/0/180] [16066.637711] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [16066.691666] LNet: Added LNI 192.168.202.106@tcp40 [8/256/0/180] [16076.015316] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [16076.055630] LNet: Added LNI 192.168.202.106@tcp41 [8/256/0/180] [16088.605937] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [16088.648182] LNet: Added LNI 192.168.202.106@tcp42 [8/256/0/180] [16098.724293] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [16098.768874] LNet: Added LNI 192.168.202.106@tcp43 [8/256/0/180] [16109.571784] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [16109.599874] LNet: Added LNI 192.168.202.106@tcp44 [8/256/0/180] [16122.405618] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [16122.455206] LNet: Added LNI 192.168.202.106@tcp45 [8/256/0/180] [16134.070440] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [16134.113293] LNet: Added LNI 192.168.202.106@tcp46 [8/256/0/180] [16143.894434] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [16143.949515] LNet: Added LNI 192.168.202.106@tcp47 [8/256/0/180] [16154.490132] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [16154.530477] LNet: Added LNI 192.168.202.106@tcp48 [8/256/0/180] [16165.260839] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [16165.292827] LNet: Added LNI 192.168.202.106@tcp49 [8/256/0/180] [16174.071893] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [16174.100653] LNet: Added LNI 192.168.202.106@tcp50 [8/256/0/180] [16203.746077] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16203.750508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16203.762095] Lustre: Skipped 1 previous similar message [16203.775369] Lustre: Skipped 2 previous similar messages [16208.001188] Lustre: server umount lustre-MDT0000 complete [16212.615373] LustreError: 475199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789419831 with bad export cookie 11440164269393489093 [16212.622871] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16212.638785] LustreError: 475199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16213.148811] Lustre: server umount lustre-MDT0001 complete [16230.367651] Lustre: 472501:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789419833/real 1789419833] req@0000000003db0990 x1876342176164224/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789419849 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16230.412323] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16230.429012] Lustre: Skipped 1 previous similar message [16233.131410] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [16234.465288] Lustre: 472502:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789419837/real 1789419837] req@000000003636ee37 x1876342176164352/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789419853 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16238.563050] Lustre: 472503:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789419841/real 1789419841] req@000000004d87f3c0 x1876342176164608/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789419857 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16243.679173] Lustre: 472501:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789419846/real 1789419846] req@0000000094e50afb x1876342176164736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789419862 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16246.395300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16257.605456] LustreError: 475193:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16257.640668] LustreError: 475193:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [16257.764372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16258.734420] LustreError: 475194:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16258.746177] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16260.786550] LustreError: 475193:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16263.502526] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [16266.225832] LustreError: 475194:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16271.348709] LustreError: 475193:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16275.491517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16276.253859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16281.027463] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [16281.588534] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16290.288164] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16298.689649] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16312.155419] LustreError: 499484:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255746 [16312.161517] LustreError: 499484:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255746 [16322.529896] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16322.536054] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16322.551765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16322.558129] Lustre: Skipped 1 previous similar message [16322.573859] Lustre: Skipped 1 previous similar message [16327.649326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16327.653767] Lustre: Skipped 1 previous similar message [16328.809165] Lustre: server umount lustre-OST0000 complete [16332.768589] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16332.776309] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16332.789686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16332.806304] Lustre: Skipped 1 previous similar message [16332.827867] Lustre: Skipped 1 previous similar message [16337.902449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16337.917692] Lustre: Skipped 1 previous similar message [16339.233632] Lustre: server umount lustre-MDT0000 complete [16343.009643] LustreError: 499489:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16343.046450] LustreError: 499489:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [16343.459239] LustreError: 473221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789419962 with bad export cookie 11440164269393491830 [16343.460599] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16343.477158] LustreError: 473221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16344.186344] Lustre: server umount lustre-MDT0001 complete [16357.801788] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [16360.810355] Key type lgssc unregistered [16361.209405] LNet: 502294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16361.215912] LNetError: 502294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16361.222612] LNet: Removed LNI 192.168.202.106@tcp [16361.224572] LNet: Removed LNI 192.168.202.106@tcp10 [16361.228723] LNet: Removed LNI 192.168.202.106@tcp11 [16361.230393] LNet: Removed LNI 192.168.202.106@tcp12 [16361.233446] LNet: Removed LNI 192.168.202.106@tcp13 [16361.235629] LNet: Removed LNI 192.168.202.106@tcp14 [16361.241565] LNet: Removed LNI 192.168.202.106@tcp15 [16361.244304] LNet: Removed LNI 192.168.202.106@tcp16 [16361.246984] LNet: Removed LNI 192.168.202.106@tcp17 [16361.249482] LNet: Removed LNI 192.168.202.106@tcp18 [16361.253469] LNet: Removed LNI 192.168.202.106@tcp19 [16361.255922] LNet: Removed LNI 192.168.202.106@tcp20 [16361.259619] LNet: Removed LNI 192.168.202.106@tcp21 [16361.262558] LNet: Removed LNI 192.168.202.106@tcp22 [16361.266565] LNet: Removed LNI 192.168.202.106@tcp23 [16361.269438] LNet: Removed LNI 192.168.202.106@tcp24 [16361.271992] LNet: Removed LNI 192.168.202.106@tcp25 [16361.275915] LNet: Removed LNI 192.168.202.106@tcp26 [16361.279160] LNet: Removed LNI 192.168.202.106@tcp27 [16361.282719] LNet: Removed LNI 192.168.202.106@tcp28 [16361.285586] LNet: Removed LNI 192.168.202.106@tcp29 [16361.290932] LNet: Removed LNI 192.168.202.106@tcp30 [16361.294635] LNet: Removed LNI 192.168.202.106@tcp31 [16361.299568] LNet: Removed LNI 192.168.202.106@tcp32 [16361.302518] LNet: Removed LNI 192.168.202.106@tcp33 [16361.304885] LNet: Removed LNI 192.168.202.106@tcp34 [16361.307618] LNet: Removed LNI 192.168.202.106@tcp35 [16361.312911] LNet: Removed LNI 192.168.202.106@tcp36 [16361.316423] LNet: Removed LNI 192.168.202.106@tcp37 [16361.320377] LNet: Removed LNI 192.168.202.106@tcp38 [16361.322908] LNet: Removed LNI 192.168.202.106@tcp39 [16361.325967] LNet: Removed LNI 192.168.202.106@tcp40 [16361.332411] LNet: Removed LNI 192.168.202.106@tcp41 [16361.340872] LNet: Removed LNI 192.168.202.106@tcp42 [16361.343480] LNet: Removed LNI 192.168.202.106@tcp43 [16361.347177] LNet: Removed LNI 192.168.202.106@tcp44 [16361.350251] LNet: Removed LNI 192.168.202.106@tcp45 [16361.352813] LNet: Removed LNI 192.168.202.106@tcp46 [16361.355732] LNet: Removed LNI 192.168.202.106@tcp47 [16361.360264] LNet: Removed LNI 192.168.202.106@tcp48 [16361.365303] LNet: Removed LNI 192.168.202.106@tcp49 [16361.379209] LNet: Removed LNI 192.168.202.106@tcp50 [16362.260303] Key type .llcrypt unregistered [16362.262339] Key type ._llcrypt unregistered [16386.388558] Key type ._llcrypt registered [16386.397543] Key type .llcrypt registered [16386.557579] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [16408.711062] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [16410.443251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16410.474320] alg: No test for adler32 (adler32-zlib) [16411.725131] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [16412.027573] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [16413.801678] Key type lgssc registered [16415.713749] Lustre: Echo OBD driver; http://www.lustre.org/ [16466.286901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16466.356984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16467.870689] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16467.929994] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16468.170515] Lustre: lustre-MDT0000: new disk, initializing [16468.416228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16468.448806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16474.754129] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [16490.083471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16490.248816] Lustre: 505493:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16490.336429] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16490.348300] Lustre: Skipped 1 previous similar message [16490.514185] Lustre: lustre-MDT0001: new disk, initializing [16490.662312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16490.731981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16490.741280] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16495.853413] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [16505.386481] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16510.623675] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16518.640661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16518.960848] Lustre: *** cfs_fail_loc=90f, val=0*** [16519.129773] Lustre: lustre-OST0000: new disk, initializing [16519.135844] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16519.148827] Lustre: 507509:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16519.275833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16526.009651] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [16526.418028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16526.424760] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16526.554807] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16535.545966] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16547.296638] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16547.300234] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16547.307041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16547.335124] Lustre: Skipped 1 previous similar message [16549.659856] Lustre: server umount lustre-OST0000 complete [16557.024474] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16557.031142] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16557.043580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16557.046155] Lustre: Skipped 1 previous similar message [16560.437343] Lustre: server umount lustre-MDT0000 complete [16564.624345] LustreError: 505486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789420183 with bad export cookie 13505568839773000752 [16564.632618] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16564.639683] LustreError: 505486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16565.082437] Lustre: server umount lustre-MDT0001 complete [16578.312358] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [16581.755257] Key type lgssc unregistered [16582.166400] LNet: 509414:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16582.177345] LNetError: 509414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16583.206559] LNet: Removed LNI 192.168.202.106@tcp [16584.310414] Key type .llcrypt unregistered [16584.313067] Key type ._llcrypt unregistered [16610.587596] Key type ._llcrypt registered [16610.588995] Key type .llcrypt registered [16610.772931] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [16613.562369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16613.606677] alg: No test for adler32 (adler32-zlib) [16615.142633] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [16615.674672] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [16617.578147] Key type lgssc registered [16619.652769] Lustre: Echo OBD driver; http://www.lustre.org/ [16627.916323] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16627.945683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16629.565405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16634.515528] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [16634.852508] LustreError: 510681:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16634.898777] LustreError: 510681:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [16638.946736] LustreError: 510682:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16644.077517] LustreError: 510681:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16644.452680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16645.012719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16650.541571] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [16660.213173] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16667.543240] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16675.404571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16675.933412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16683.322994] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [16694.025328] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16715.408423] loop0: detected capacity change from 0 to 204800000 [16715.746452] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16715.773800] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16715.797874] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16723.145971] loop0: detected capacity change from 0 to 204800000 [16723.321720] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16723.327876] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16723.336209] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16731.212972] loop0: detected capacity change from 0 to 204800000 [16740.630082] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16740.810493] Lustre: fs15246-OST0000: new disk, initializing [16740.814727] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16740.828808] Lustre: 514301:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16740.947960] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16751.150698] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [16762.105149] loop1: detected capacity change from 0 to 204800000 [16768.629979] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16768.815344] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16768.877350] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16768.940923] Lustre: fs15246-MDT0000: new disk, initializing [16769.154361] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16769.237529] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16769.401603] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16770.469632] Lustre: 515425:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000005197d35d x1876343111342208/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789420405 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16774.745991] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [16779.554525] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16779.561399] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16779.607194] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16806.884182] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16806.894569] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16811.707282] Lustre: server umount fs15246-MDT0000 complete [16812.013555] LustreError: 510675:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16820.209195] LustreError: 510678:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16820.235582] LustreError: 510678:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [16821.208968] Lustre: server umount fs15246-OST0000 complete [16834.023876] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16834.024477] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16834.028207] LustreError: Skipped 1 previous similar message [16834.029066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16834.040129] Lustre: Skipped 1 previous similar message [16834.065613] Lustre: Skipped 1 previous similar message [16835.560635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16835.563689] Lustre: Skipped 1 previous similar message [16838.618670] Lustre: server umount lustre-OST0000 complete [16844.770411] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16844.787283] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16844.820454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16848.710199] Lustre: server umount lustre-MDT0000 complete [16850.921232] LustreError: 510678:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16850.939913] LustreError: 510678:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [16853.378097] LustreError: 510663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789420472 with bad export cookie 10563994516157242258 [16853.383147] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16853.393142] LustreError: 510663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16854.258588] Lustre: server umount lustre-MDT0001 complete [16866.158589] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [16870.551751] Key type lgssc unregistered [16871.015829] LNet: 518523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16871.067602] LNetError: 518523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16872.173924] LNet: Removed LNI 192.168.202.106@tcp [16873.406135] Key type .llcrypt unregistered [16873.407900] Key type ._llcrypt unregistered [16894.631066] Key type ._llcrypt registered [16894.633624] Key type .llcrypt registered [16894.767891] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [16896.589238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16896.618710] alg: No test for adler32 (adler32-zlib) [16897.914778] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [16898.320062] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [16900.199239] Key type lgssc registered [16901.805936] Lustre: Echo OBD driver; http://www.lustre.org/ [16947.163282] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [16959.628955] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [17007.213846] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17007.227263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17008.498657] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [17008.533295] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [17008.605719] Lustre: lustre-MDT0000: new disk, initializing [17008.714613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17008.743600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [17014.018217] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [17025.210637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17025.363869] Lustre: 522988:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [17025.411550] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [17025.414382] Lustre: Skipped 1 previous similar message [17025.507183] Lustre: lustre-MDT0001: new disk, initializing [17025.634110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17025.675173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [17025.694449] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [17030.524866] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [17039.399993] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17044.987891] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17051.938644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17052.265789] Lustre: lustre-OST0000: new disk, initializing [17052.269729] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [17052.275619] Lustre: 524958:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17052.451312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17059.035964] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [17059.978270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [17059.995560] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [17060.105948] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [17068.205933] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17072.785849] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [17072.931576] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [17077.236781] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [17077.498245] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [17080.802639] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17080.803625] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [17080.840604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17080.865325] Lustre: Skipped 1 previous similar message [17085.464437] Lustre: server umount lustre-OST0000 complete [17091.042392] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17091.084254] Lustre: Skipped 1 previous similar message [17091.099967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17091.108402] Lustre: Skipped 1 previous similar message [17095.991484] Lustre: server umount lustre-MDT0000 complete [17100.882548] LustreError: 523796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789420720 with bad export cookie 3316318984334246255 [17100.903933] LustreError: 523796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [17100.905722] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17101.148791] Lustre: server umount lustre-MDT0001 complete [17128.632475] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [17141.931344] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [17181.240851] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16864 sec ========= 17:19:58 (1789420798) [17183.076481] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 17:20:00 (1789420800) === [17187.073589] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 17:20:04 (1789420804) === [17232.139640] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [17235.118448] Key type lgssc unregistered [17235.405669] LNet: 531603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17235.415778] LNetError: 531603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17236.473692] LNet: Removed LNI 192.168.202.106@tcp [17237.024337] Key type .llcrypt unregistered [17237.025617] Key type ._llcrypt unregistered