[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 512587258 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.002390] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007026] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008016] pid_max: default: 32768 minimum: 301 [ 0.009167] LSM: Security Framework initializing [ 0.010072] Yama: becoming mindful. [ 0.012046] SELinux: Initializing. [ 0.013099] *** VALIDATE selinux *** [ 0.021783] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025789] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026165] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027102] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028107] *** VALIDATE tmpfs *** [ 0.029417] *** VALIDATE proc *** [ 0.031204] *** VALIDATE cgroup *** [ 0.032011] *** VALIDATE cgroup2 *** [ 0.033268] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034149] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036026] Spectre V2 : User space: Vulnerable [ 0.037008] Speculative Store Bypass: Vulnerable [ 0.040133] debug: unmapping init [mem 0xffffffff86c59000-0xffffffff86c60fff] [ 0.042205] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043743] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044018] ... version: 2 [ 0.045010] ... bit width: 48 [ 0.045908] ... generic registers: 4 [ 0.046013] ... value mask: 0000ffffffffffff [ 0.047013] ... max period: 00007fffffffffff [ 0.048015] ... fixed-purpose events: 3 [ 0.049014] ... event mask: 000000070000000f [ 0.050315] rcu: Hierarchical SRCU implementation. [ 0.052425] smp: Bringing up secondary CPUs ... [ 0.053610] x86: Booting SMP configuration: [ 0.054031] .... node #0, CPUs: #1 #2 #3 [ 0.058189] smp: Brought up 1 node, 4 CPUs [ 0.060019] smpboot: Max logical packages: 1 [ 0.061015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.235000] node 0 deferred pages initialised in 171ms [ 0.239095] devtmpfs: initialized [ 0.240632] x86/mm: Memory block size: 128MB [ 0.244441] gcov: version magic: 0x41383552 [ 0.248495] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.253115] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.256437] pinctrl core: initialized pinctrl subsystem [ 0.258300] [ 0.258866] ************************************************************* [ 0.262026] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.264018] ** ** [ 0.267022] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.270026] ** ** [ 0.273018] ** This means that this kernel is built to expose internal ** [ 0.275015] ** IOMMU data structures, which may compromise security on ** [ 0.277020] ** your system. ** [ 0.280021] ** ** [ 0.283020] ** If you see this message and you are not debugging the ** [ 0.285016] ** kernel, report this immediately to your vendor! ** [ 0.287018] ** ** [ 0.290035] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.292021] ************************************************************* [ 0.295646] NET: Registered protocol family 16 [ 0.298627] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.301087] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.304099] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.307603] cpuidle: using governor menu [ 0.309610] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.312631] PCI: Using configuration type 1 for base access [ 0.315144] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.326147] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.330035] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.334076] cryptd: max_cpu_qlen set to 1000 [ 0.339235] ACPI: Added _OSI(Module Device) [ 0.342053] ACPI: Added _OSI(Processor Device) [ 0.343021] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.345013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.350164] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.356602] ACPI: Interpreter enabled [ 0.359098] ACPI: PM: (supports S0 S3 S4 S5) [ 0.361022] ACPI: Using IOAPIC for interrupt routing [ 0.363184] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.369000] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.381481] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.385066] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.388028] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.392103] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.397492] acpiphp: Slot [2] registered [ 0.399123] acpiphp: Slot [5] registered [ 0.401126] acpiphp: Slot [6] registered [ 0.402098] acpiphp: Slot [7] registered [ 0.404108] acpiphp: Slot [8] registered [ 0.405105] acpiphp: Slot [9] registered [ 0.407243] acpiphp: Slot [10] registered [ 0.409221] acpiphp: Slot [3] registered [ 0.410124] acpiphp: Slot [4] registered [ 0.412170] acpiphp: Slot [11] registered [ 0.414178] acpiphp: Slot [12] registered [ 0.415100] acpiphp: Slot [13] registered [ 0.417091] acpiphp: Slot [14] registered [ 0.418088] acpiphp: Slot [15] registered [ 0.420143] acpiphp: Slot [16] registered [ 0.422200] acpiphp: Slot [17] registered [ 0.425176] acpiphp: Slot [18] registered [ 0.427179] acpiphp: Slot [19] registered [ 0.429165] acpiphp: Slot [20] registered [ 0.431161] acpiphp: Slot [21] registered [ 0.433181] acpiphp: Slot [22] registered [ 0.434209] acpiphp: Slot [23] registered [ 0.436163] acpiphp: Slot [24] registered [ 0.438138] acpiphp: Slot [25] registered [ 0.440142] acpiphp: Slot [26] registered [ 0.441148] acpiphp: Slot [27] registered [ 0.443162] acpiphp: Slot [28] registered [ 0.444133] acpiphp: Slot [29] registered [ 0.446179] acpiphp: Slot [30] registered [ 0.448157] acpiphp: Slot [31] registered [ 0.450101] PCI host bridge to bus 0000:00 [ 0.452021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.454024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.456020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.458019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.460021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.463024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.464178] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.467047] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.469269] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.478691] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.483528] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.485020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.487017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.489016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.491520] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.494879] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.498067] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.501917] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.507024] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.521037] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.527022] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.535936] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.555016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.570015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.601021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.617221] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.626018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.635016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.656018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.667074] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.673017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.679016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.695023] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.708853] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.718017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.728015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.748019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.760827] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.773122] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.782018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.804018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.816222] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.825034] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.837025] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.859029] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.876229] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.880425] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.883378] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.887401] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.891280] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.897193] iommu: Default domain type: Passthrough [ 0.900543] SCSI subsystem initialized [ 0.902241] ACPI: bus type USB registered [ 0.904125] usbcore: registered new interface driver usbfs [ 0.906176] usbcore: registered new interface driver hub [ 0.909119] usbcore: registered new device driver usb [ 0.912202] pps_core: LinuxPPS API ver. 1 registered [ 0.915015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.920066] PTP clock support registered [ 0.924043] EDAC MC: Ver: 3.0.0 [ 0.925753] PCI: Using ACPI for IRQ routing [ 0.926948] NetLabel: Initializing [ 0.927011] NetLabel: domain hash size = 128 [ 0.928009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.929153] NetLabel: unlabeled traffic allowed by default [ 0.932188] vgaarb: loaded [ 0.934433] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.936013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.942781] clocksource: Switched to clocksource kvm-clock [ 1.064223] VFS: Disk quotas dquot_6.6.0 [ 1.065474] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.068055] *** VALIDATE ramfs *** [ 1.069097] *** VALIDATE hugetlbfs *** [ 1.070615] pnp: PnP ACPI init [ 1.072872] pnp: PnP ACPI: found 6 devices [ 1.093737] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.098313] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.100350] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.102022] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.103840] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.106246] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.108911] NET: Registered protocol family 2 [ 1.111373] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.116296] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.124796] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.130477] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.133971] TCP: Hash tables configured (established 65536 bind 65536) [ 1.136995] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.140133] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.143455] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.146533] NET: Registered protocol family 1 [ 1.149437] RPC: Registered named UNIX socket transport module. [ 1.152459] RPC: Registered udp transport module. [ 1.155422] RPC: Registered tcp transport module. [ 1.158448] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.162666] NET: Registered protocol family 44 [ 1.164683] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.168430] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.171174] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.176232] PCI: CLS 0 bytes, default 64 [ 1.178199] Unpacking initramfs... [ 2.639967] debug: unmapping init [mem 0xffff8d7b3cc54000-0xffff8d7b3ffbffff] [ 2.645245] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.648431] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.653021] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.166885] Initialise system trusted keyrings [ 3.169100] Key type blacklist registered [ 3.173829] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.186815] zbud: loaded [ 3.190052] *** VALIDATE nfs *** [ 3.191456] *** VALIDATE nfs4 *** [ 3.193202] pstore: using deflate compression [ 3.197611] Platform Keyring initialized [ 3.308874] NET: Registered protocol family 38 [ 3.311055] Key type asymmetric registered [ 3.313049] Asymmetric key parser 'x509' registered [ 3.315196] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.318648] io scheduler mq-deadline registered [ 3.320358] io scheduler kyber registered [ 3.322117] io scheduler bfq registered [ 3.324559] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.327891] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.330989] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.334203] ACPI: Power Button [PWRF] [ 3.339990] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.346832] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.361691] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.370407] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.391846] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.419599] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.449497] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.455188] Non-volatile memory driver v1.3 [ 3.457275] Linux agpgart interface v0.103 [ 3.495146] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.498531] vda: detected capacity change from 0 to 74698752 [ 3.515549] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.519111] vdb: detected capacity change from 0 to 1073741824 [ 3.534625] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.538351] vdc: detected capacity change from 0 to 2621440000 [ 3.558098] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.561410] vdd: detected capacity change from 0 to 2621440000 [ 3.583591] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.587141] vde: detected capacity change from 0 to 4294967296 [ 3.603267] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.606232] vdf: detected capacity change from 0 to 4294967296 [ 3.613880] libphy: Fixed MDIO Bus: probed [ 3.620377] usbcore: registered new interface driver usbserial_generic [ 3.623764] usbserial: USB Serial support registered for generic [ 3.626953] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.632200] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.634343] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.636625] mousedev: PS/2 mouse device common for all mice [ 3.640225] rtc_cmos 00:05: RTC can wake from S4 [ 3.643544] rtc_cmos 00:05: registered as rtc0 [ 3.645428] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.646258] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.652851] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.655088] intel_pstate: CPU model not supported [ 3.660266] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.665528] hid: raw HID events driver (C) Jiri Kosina [ 3.667821] usbcore: registered new interface driver usbhid [ 3.670370] usbhid: USB HID core driver [ 3.672325] drop_monitor: Initializing network drop monitor service [ 3.675277] Initializing XFRM netlink socket [ 3.677489] NET: Registered protocol family 10 [ 3.681087] Segment Routing with IPv6 [ 3.682694] NET: Registered protocol family 17 [ 3.685322] mpls_gso: MPLS GSO support [ 3.692108] RAS: Correctable Errors collector initialized. [ 3.694699] AVX version of gcm_enc/dec engaged. [ 3.696265] AES CTR mode by8 optimization enabled [ 3.785966] sched_clock: Marking stable (3785874954, 0)->(4739086457, -953211503) [ 3.790132] registered taskstats version 1 [ 3.792387] Loading compiled-in X.509 certificates [ 3.795043] zswap: loaded using pool lzo/zbud [ 3.831584] Key type big_key registered [ 3.847870] Key type encrypted registered [ 3.850400] ima: No TPM chip found, activating TPM-bypass! [ 3.853350] ima: Allocated hash algorithm: sha1 [ 3.855850] ima: No architecture policies found [ 3.857876] evm: Initialising EVM extended attributes: [ 3.860380] evm: security.selinux [ 3.861818] evm: security.ima [ 3.862973] evm: security.capability [ 3.864535] evm: HMAC attrs: 0x1 [ 3.867493] rtc_cmos 00:05: setting system clock to 2026-09-03 13:09:39 UTC (1788440979) [ 3.875215] debug: unmapping init [mem 0xffffffff87c03000-0xffffffff87dfffff] [ 3.878424] debug: unmapping init [mem 0xffffffff86982000-0xffffffff86c58fff] [ 3.890268] Write protecting the kernel read-only data: 28672k [ 3.894290] debug: unmapping init [mem 0xffffffff85003000-0xffffffff851fffff] [ 3.897557] debug: unmapping init [mem 0xffffffff85914000-0xffffffff859fffff] [ 3.940347] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.951261] systemd[1]: Detected virtualization kvm. [ 3.953692] systemd[1]: Detected architecture x86-64. [ 3.956410] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.986235] systemd[1]: No hostname configured. [ 3.988264] systemd[1]: Set hostname to . [ 3.990811] random: systemd: uninitialized urandom read (16 bytes read) [ 3.993851] systemd[1]: Initializing machine ID from random generator. [ 4.150575] random: systemd: uninitialized urandom read (16 bytes read) [ 4.154294] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.159134] random: systemd: uninitialized urandom read (16 bytes read) [ 4.161890] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.167880] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Swap. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.860583] device-mapper: uevent: version 1.0.3 [ 4.863161] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.621907] virtio_net virtio0 ens2: renamed from eth0 [ 5.628862] random: fast init done [ 5.814886] scsi host0: ata_piix [ 5.859121] scsi host1: ata_piix [ 5.861497] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.864044] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.598033] dracut-initqueue[577]: RTNETLINK answers: File exists [ 10.372940] random: crng init done [ 10.374433] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.989164] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.285893] printk: systemd: 23 output lines suppressed due to ratelimiting [ 12.592473] SELinux: Disabled at runtime. [ 12.664881] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.674459] systemd[1]: Detected virtualization kvm. [ 12.676513] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.240056] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.244554] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.253421] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.258412] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.262243] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.269944] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.275731] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Paths. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting udev Coldplug all Devices... [ 13.445700] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.777896] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.247138] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.266306] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.490155] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.503884] EDAC sbridge: Ver: 1.1.2 [ 16.142870] Key type dns_resolver registered [ 16.475109] NFS: Registering the id_resolver key type [ 16.477750] Key type id_resolver registered [ 16.479373] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg440-server login: [ 70.543911] hrtimer: interrupt took 4957687 ns [ 70.808309] libcfs: loading out-of-tree module taints kernel. [ 70.933315] Key type ._llcrypt registered [ 70.935800] Key type .llcrypt registered [ 71.103758] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 90.550637] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 92.129194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 92.145525] alg: No test for adler32 (adler32-zlib) [ 93.565714] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 94.642284] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 96.575381] Key type lgssc registered [ 98.655823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 116.767631] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 164.527452] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 178.986461] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 179.031524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 180.360441] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 180.397427] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 180.547600] Lustre: lustre-MDT0000: new disk, initializing [ 180.709985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 180.735503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 185.945055] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 200.673962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 200.807232] Lustre: 6523:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 200.889686] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 200.895519] Lustre: Skipped 1 previous similar message [ 201.019720] Lustre: lustre-MDT0001: new disk, initializing [ 201.107927] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 201.160149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 201.179168] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 205.846432] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 211.141626] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 221.889243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 222.134517] Lustre: lustre-OST0000: new disk, initializing [ 222.138996] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 222.149152] Lustre: 8458:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 222.243281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 227.401495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 227.409453] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 227.488571] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 228.313495] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 244.777726] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 244.966328] Lustre: lustre-OST0001: new disk, initializing [ 244.969842] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 244.974888] Lustre: 9533:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 245.032220] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 252.060274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 252.085794] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 252.211802] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 252.439609] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 266.893667] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 274.870041] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 283.773403] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing check_logdir /tmp/testlogs/ [ 290.536380] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing yml_node [ 296.888857] Lustre: DEBUG MARKER: Client: 2.17.56.51 [ 300.291184] Lustre: DEBUG MARKER: MDS: 2.17.56.51 [ 303.970519] Lustre: DEBUG MARKER: OSS: 2.17.56.51 [ 306.798583] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 09:14:40 EDT 2026 [ 329.406662] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 331.163664] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 349.667524] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 349.670921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 349.689601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 354.788293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 354.797752] Lustre: Skipped 3 previous similar messages [ 355.029972] Lustre: server umount lustre-MDT0000 complete [ 359.453447] LustreError: 10348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441335 with bad export cookie 12306796129056900242 [ 359.467229] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 359.477146] LustreError: 10348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 359.905580] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 359.924888] Lustre: Skipped 3 previous similar messages [ 359.940355] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 359.947254] Lustre: Skipped 4 previous similar messages [ 365.043887] LustreError: 6528:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 365.044434] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 365.076100] LustreError: 6528:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 365.094699] Lustre: Skipped 1 previous similar message [ 366.187516] Lustre: server umount lustre-MDT0001 complete [ 370.852277] Lustre: server umount lustre-OST0000 complete [ 375.741086] Lustre: server umount lustre-OST0001 complete [ 384.344511] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 395.921733] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 437.752465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 438.072386] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 438.119320] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 438.281841] Lustre: lustre-MDT0000: new disk, initializing [ 438.392250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 438.438854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 443.931181] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 456.576870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 456.805869] Lustre: 15090:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 456.923365] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 456.930781] Lustre: Skipped 1 previous similar message [ 457.005200] Lustre: lustre-MDT0001: new disk, initializing [ 457.099751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 457.142268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 457.162279] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 462.566846] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 472.998228] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 480.067670] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 488.399581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 488.793254] Lustre: lustre-OST0000: new disk, initializing [ 488.799567] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 488.809145] Lustre: 17060:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 488.970132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 490.775817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 490.795082] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 490.912775] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 496.074257] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 507.428153] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 513.087938] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 513.317716] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 519.196890] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 519.500033] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 526.820717] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 526.824994] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 526.825141] LustreError: Skipped 1 previous similar message [ 526.833172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 526.839504] Lustre: Skipped 2 previous similar messages [ 526.869425] Lustre: Skipped 1 previous similar message [ 528.211421] Lustre: server umount lustre-OST0000 complete [ 533.984987] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 534.007177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 537.060689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 537.063028] Lustre: Skipped 1 previous similar message [ 538.721435] Lustre: server umount lustre-MDT0000 complete [ 542.185595] LustreError: 15103:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 542.209690] LustreError: 15103:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 543.369756] LustreError: 15083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441519 with bad export cookie 12306796129056901775 [ 543.385842] LustreError: 15083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 543.386428] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 543.849838] Lustre: server umount lustre-MDT0001 complete [ 548.977300] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 09:18:43 (1788441523) [ 552.072141] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 554.372484] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 09:18:48 (1788441528) [ 561.803516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 562.308272] LustreError: 19469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 562.377433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 566.868266] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 567.782533] LustreError: 19470:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 571.876404] LustreError: 19469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 576.312503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 582.371364] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 592.016168] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 599.008211] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 607.458252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 615.530524] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 625.019585] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 647.140135] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 647.159183] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 647.175845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 647.182081] Lustre: Skipped 2 previous similar messages [ 647.206676] Lustre: Skipped 2 previous similar messages [ 649.865641] Lustre: server umount lustre-OST0000 complete [ 657.383644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 657.394613] Lustre: Skipped 1 previous similar message [ 659.869954] Lustre: server umount lustre-MDT0000 complete [ 662.502393] LustreError: 20421:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 662.532340] LustreError: 20421:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 663.876502] LustreError: 22074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441639 with bad export cookie 12306796129056902657 [ 663.881385] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 663.901849] LustreError: 22074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 664.321783] Lustre: server umount lustre-MDT0001 complete [ 675.637705] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 678.844637] Key type lgssc unregistered [ 679.167028] LNet: 23032:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 679.173407] LNetError: 23032:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 679.196661] LNet: Removed LNI 192.168.204.140@tcp [ 680.242298] Key type .llcrypt unregistered [ 680.247515] Key type ._llcrypt unregistered [ 706.169639] Key type ._llcrypt registered [ 706.173540] Key type .llcrypt registered [ 706.340764] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 707.766081] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 707.807437] alg: No test for adler32 (adler32-zlib) [ 709.147796] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 709.469884] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 711.224681] Key type lgssc registered [ 712.236454] Lustre: Echo OBD driver; http://www.lustre.org/ [ 721.290223] loop: module loaded [ 721.371625] loop0: detected capacity change from 0 to 204800000 [ 721.649223] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 721.658918] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 721.978569] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 728.378755] loop0: detected capacity change from 0 to 204800000 [ 728.546569] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 728.566773] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 728.581787] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 734.514642] loop0: detected capacity change from 0 to 204800000 [ 742.182285] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 742.211870] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 743.457201] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 743.516650] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 743.586256] Lustre: lustre-MDT0000: new disk, initializing [ 743.655654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 743.684331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 749.495745] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 759.338150] loop1: detected capacity change from 0 to 204800000 [ 765.212682] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 765.647574] Lustre: lustre-OST07c6: new disk, initializing [ 765.652287] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 765.658244] Lustre: Skipped 1 previous similar message [ 765.662427] Lustre: 26219:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 765.751915] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 771.687726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 771.695761] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 771.743041] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 772.068322] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 777.460508] Lustre: 25205:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 777.475102] Lustre: 25205:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 777.486562] Lustre: 25205:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 777.503050] Lustre: 25205:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 777.513260] Lustre: 25205:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 777.541216] Lustre: 25205:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 796.367903] Lustre: 25206:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 796.386318] Lustre: 25206:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 796.401018] Lustre: 25206:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 796.416805] Lustre: 25206:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 796.427249] Lustre: 25206:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 796.435713] Lustre: 25206:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 802.383635] Lustre: 25206:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 802.401077] Lustre: 25206:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 802.412165] Lustre: 25206:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 802.418073] Lustre: 25206:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 802.423245] Lustre: 25206:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 802.428784] Lustre: 25206:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 802.434580] Lustre: 25206:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 802.439952] Lustre: 25206:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 802.446810] Lustre: 25206:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 802.451583] Lustre: 25206:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 802.457341] Lustre: 25206:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 802.462291] Lustre: 25206:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 805.672824] Lustre: 26227:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 805.681580] Lustre: 26227:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 805.686472] Lustre: 26227:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 805.692435] Lustre: 26227:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 805.697229] Lustre: 26227:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 805.702921] Lustre: 26227:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 805.710500] Lustre: 26227:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 805.715893] Lustre: 26227:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 805.720673] Lustre: 26227:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 805.730448] Lustre: 26227:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 805.735976] Lustre: 26227:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 805.742673] Lustre: 26227:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 814.005740] Lustre: 25206:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 814.013173] Lustre: 25206:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 814.017299] Lustre: 25206:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 814.021161] Lustre: 25206:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 814.025050] Lustre: 25206:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 814.032666] Lustre: 25206:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 814.041170] Lustre: 25206:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 814.050580] Lustre: 25206:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 814.063326] Lustre: 25206:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 814.073780] Lustre: 25206:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 814.091287] Lustre: 25206:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 814.095670] Lustre: 25206:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 823.653631] Lustre: 26227:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 823.667230] Lustre: 26227:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 823.676594] Lustre: 26227:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 823.690469] Lustre: 26227:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 823.700265] Lustre: 26227:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 823.714707] Lustre: 26227:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 823.724712] Lustre: 26227:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 823.734596] Lustre: 26227:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 823.749969] Lustre: 26227:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 823.761627] Lustre: 26227:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 823.779723] Lustre: 26227:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 823.785922] Lustre: 26227:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 868.832321] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 868.863529] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 873.950638] Lustre: server umount lustre-OST07c6 complete [ 883.616369] Lustre: server umount lustre-MDT0000 complete [ 901.615292] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 904.408798] Key type lgssc unregistered [ 904.686315] LNet: 28786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 904.692162] LNetError: 28786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 904.710248] LNet: Removed LNI 192.168.204.140@tcp [ 905.551574] Key type .llcrypt unregistered [ 905.553961] Key type ._llcrypt unregistered [ 926.585563] Key type ._llcrypt registered [ 926.588661] Key type .llcrypt registered [ 926.726648] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 927.990287] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 928.015225] alg: No test for adler32 (adler32-zlib) [ 929.272775] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 929.534368] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 931.211356] Key type lgssc registered [ 932.315392] Lustre: Echo OBD driver; http://www.lustre.org/ [ 941.650259] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 941.674486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 943.243226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 948.295238] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 952.802522] LustreError: 30059:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 952.838111] LustreError: 30059:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 957.840788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 957.924848] LustreError: 30060:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 958.381296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 962.314256] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 969.908178] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 976.587047] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 984.191036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 984.764357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 987.841910] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 992.498117] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1000.738278] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1023.968710] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1023.976196] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1023.976993] LustreError: Skipped 1 previous similar message [ 1023.980429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1023.990023] Lustre: Skipped 1 previous similar message [ 1026.289695] Lustre: server umount lustre-OST0000 complete [ 1034.210788] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1034.211928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1034.224626] Lustre: Skipped 1 previous similar message [ 1034.233145] Lustre: Skipped 2 previous similar messages [ 1036.782520] Lustre: server umount lustre-MDT0000 complete [ 1039.334827] LustreError: 31112:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1039.359073] LustreError: 31112:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1040.929500] LustreError: 30042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442016 with bad export cookie 10745892418293308777 [ 1040.930991] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1040.937075] LustreError: 30042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1041.370695] Lustre: server umount lustre-MDT0001 complete [ 1053.179855] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 1056.038486] Key type lgssc unregistered [ 1056.270368] LNet: 33796:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1056.278776] LNetError: 33796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1056.297601] LNet: Removed LNI 192.168.204.140@tcp [ 1057.105377] Key type .llcrypt unregistered [ 1057.108679] Key type ._llcrypt unregistered [ 1078.700337] Key type ._llcrypt registered [ 1078.706549] Key type .llcrypt registered [ 1078.856709] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 1080.368807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1080.454953] alg: No test for adler32 (adler32-zlib) [ 1081.765962] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1081.947834] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 1083.631242] Key type lgssc registered [ 1085.278958] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1093.861424] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1093.872358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1095.360255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1100.056925] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1104.877757] LustreError: 35067:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1104.895059] LustreError: 35067:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1108.826432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1109.228641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1113.752286] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1121.679372] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1127.747571] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1135.852106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1136.165174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1140.283529] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1143.411599] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1152.207400] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1163.752368] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1163.757908] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1163.768383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1168.871427] LustreError: 36889:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1168.892355] LustreError: 36889:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1169.028444] Lustre: server umount lustre-OST0000 complete [ 1173.998313] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1174.004492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1174.011110] Lustre: Skipped 2 previous similar messages [ 1174.021932] Lustre: Skipped 2 previous similar messages [ 1179.105565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1179.113549] Lustre: Skipped 1 previous similar message [ 1184.224779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1184.233186] Lustre: Skipped 1 previous similar message [ 1187.300123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1187.657984] Lustre: server umount lustre-MDT0000 complete [ 1189.347899] LustreError: 36106:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1189.371296] LustreError: 36106:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1191.766970] LustreError: 37848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442167 with bad export cookie 17707660440950765061 [ 1191.781275] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1191.793132] LustreError: 37848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1192.220467] Lustre: server umount lustre-MDT0001 complete [ 1205.176548] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 1208.270885] Key type lgssc unregistered [ 1208.692910] LNet: 38603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1208.708263] LNetError: 38603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1208.722377] LNet: Removed LNI 192.168.204.140@tcp [ 1209.799185] Key type .llcrypt unregistered [ 1209.802774] Key type ._llcrypt unregistered [ 1235.783277] Key type ._llcrypt registered [ 1235.785777] Key type .llcrypt registered [ 1235.885987] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 1237.623917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1237.664374] alg: No test for adler32 (adler32-zlib) [ 1238.805078] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1239.060976] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 1240.744310] Key type lgssc registered [ 1242.150150] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1251.091717] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1251.106537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1252.795209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1257.979829] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1263.082699] LustreError: 40021:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1263.111270] LustreError: 40021:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1268.056441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1268.192119] LustreError: 40020:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1268.452812] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1273.445966] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1281.782840] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1287.775460] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1295.410458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1295.922721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1297.995283] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1303.505477] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1313.148852] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1321.583797] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.40@tcp (stopping) [ 1324.001249] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1324.018919] Lustre: Skipped 1 previous similar message [ 1324.021479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1324.038034] Lustre: Skipped 2 previous similar messages [ 1326.675431] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.40@tcp (stopping) [ 1327.714339] Lustre: server umount lustre-MDT0000 complete [ 1329.131953] LustreError: 41025:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1329.157250] LustreError: 41025:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1331.807036] LustreError: 40015:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1332.085884] LustreError: 40001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442307 with bad export cookie 4608158048678367873 [ 1332.099374] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1332.100654] LustreError: 40001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1334.243249] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1334.261278] Lustre: Skipped 1 previous similar message [ 1334.268343] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1336.920122] LustreError: 40016:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1336.962566] LustreError: 40016:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1338.583822] Lustre: server umount lustre-MDT0001 complete [ 1360.546960] Lustre: server umount lustre-OST0000 complete [ 1378.297094] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 1382.206337] Key type lgssc unregistered [ 1382.463854] LNet: 43886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1382.484361] LNetError: 43886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1382.522474] LNet: Removed LNI 192.168.204.140@tcp [ 1383.673217] Key type .llcrypt unregistered [ 1383.675937] Key type ._llcrypt unregistered [ 1413.397526] Key type ._llcrypt registered [ 1413.399543] Key type .llcrypt registered [ 1413.518811] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 1414.546322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1414.591349] alg: No test for adler32 (adler32-zlib) [ 1415.812744] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1415.996722] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 1417.631606] Key type lgssc registered [ 1418.584467] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1427.644594] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1427.681856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1429.497706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1434.632635] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1439.714256] LustreError: 45303:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1439.733552] LustreError: 45303:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1444.834754] LustreError: 45304:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1445.224279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1445.747058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1451.227171] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1459.067209] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1465.398053] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1473.326549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1473.775407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1474.815570] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1480.143720] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1487.847735] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1499.766526] Lustre: lustre-OST0000: Not available for connect from 192.168.204.40@tcp (stopping) [ 1500.646801] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1500.657689] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1500.675326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1501.669164] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1501.671826] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1501.707309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1501.712314] Lustre: Skipped 1 previous similar message [ 1504.863862] Lustre: lustre-OST0000: Not available for connect from 192.168.204.40@tcp (stopping) [ 1509.979981] Lustre: lustre-OST0000: Not available for connect from 192.168.204.40@tcp (stopping) [ 1509.993755] Lustre: Skipped 2 previous similar messages [ 1511.391413] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1511.648271] Lustre: server umount lustre-OST0000 complete [ 1531.747326] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1531.754903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1531.765847] Lustre: Skipped 1 previous similar message [ 1531.796894] Lustre: Skipped 1 previous similar message [ 1545.183680] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1545.491213] Lustre: server umount lustre-MDT0000 complete [ 1547.244691] LustreError: 45304:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1547.273304] LustreError: 45304:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1549.358822] LustreError: 45286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442524 with bad export cookie 18244189557631561035 [ 1549.366848] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1549.369588] LustreError: 45286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1555.857190] Lustre: server umount lustre-MDT0001 complete [ 1568.474717] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 1571.901498] Key type lgssc unregistered [ 1572.318238] LNet: 49018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1572.325795] LNetError: 49018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1572.373371] LNet: Removed LNI 192.168.204.140@tcp [ 1573.397498] Key type .llcrypt unregistered [ 1573.399249] Key type ._llcrypt unregistered [ 1601.956474] Key type ._llcrypt registered [ 1601.959774] Key type .llcrypt registered [ 1602.125776] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 1603.437028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1603.540349] alg: No test for adler32 (adler32-zlib) [ 1604.652316] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1604.963141] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 1606.648578] Key type lgssc registered [ 1607.977724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1616.343602] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1616.361849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1617.959166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1623.058804] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1628.131623] LustreError: 50440:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1628.155631] LustreError: 50440:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1632.170914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1632.665807] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1637.954309] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1646.592063] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1653.248224] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1660.322137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1660.622574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1662.737241] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1667.524095] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1676.950415] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1684.008501] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1688.199604] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1700.796751] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-03 9h37m54s [ 1702.213558] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1704.416683] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1704.428446] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1704.460141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1705.508083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1705.515463] Lustre: Skipped 4 previous similar messages [ 1706.643186] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.40@tcp (stopping) [ 1706.650087] Lustre: Skipped 1 previous similar message [ 1710.456641] Lustre: server umount lustre-MDT0000 complete [ 1711.588272] LustreError: 50439:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1711.603519] LustreError: 50439:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1714.186881] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1717.740127] LustreError: 51167:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1717.771966] LustreError: 51167:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1722.200442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1722.337876] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1722.736900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1722.862709] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1723.773453] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1728.762415] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1732.969457] Lustre: DEBUG MARKER: Wait for df (36724) ... [ 1734.960593] Lustre: DEBUG MARKER: done [ 1739.744360] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1739.753522] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1739.772578] Lustre: Skipped 2 previous similar messages [ 1739.791640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1743.335611] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1743.341336] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1744.365732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1744.379801] Lustre: Skipped 2 previous similar messages [ 1745.624195] Lustre: server umount lustre-OST0000 complete [ 1754.599620] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1754.600604] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1754.605543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1754.638276] Lustre: Skipped 1 previous similar message [ 1764.319153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1764.547885] Lustre: server umount lustre-MDT0000 complete [ 1767.913382] LustreError: 50439:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1767.936102] LustreError: 50439:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1768.751302] LustreError: 50420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442744 with bad export cookie 674559130016423271 [ 1768.756877] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1768.775865] LustreError: 50420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1769.060863] Lustre: server umount lustre-MDT0001 complete [ 1780.031116] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 1782.963677] Key type lgssc unregistered [ 1783.353248] LNet: 55374:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1783.376855] LNetError: 55374:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1783.400295] LNet: Removed LNI 192.168.204.140@tcp [ 1784.336243] Key type .llcrypt unregistered [ 1784.338244] Key type ._llcrypt unregistered [ 1832.857613] Key type ._llcrypt registered [ 1832.860831] Key type .llcrypt registered [ 1833.008362] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 1834.881125] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1834.911538] alg: No test for adler32 (adler32-zlib) [ 1836.090772] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1836.451417] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 1838.183272] Key type lgssc registered [ 1839.688794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1848.349935] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1848.374477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1849.718240] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1849.747255] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1849.950335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1855.337287] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1866.098745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1866.303227] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1866.355699] Lustre: 57584:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1866.606956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1872.094193] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1880.419595] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1887.049280] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1894.723499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1894.948799] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1895.236044] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1902.526177] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1903.250627] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1911.563113] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1919.122814] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-03 9h41m33s [ 1920.843778] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1928.538972] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1931.730917] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1931.824983] LustreError: 57593:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1951.899857] Lustre: lustre-MDT0000: Client 0d248a40-3c55-437e-be7c-555c3ee09d58 (at 192.168.204.40@tcp) reconnecting [ 1971.927248] LustreError: 57593:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1974.790691] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1978.066197] Lustre: DEBUG MARKER: done [ 1984.994255] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1985.007966] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1985.022256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1985.505925] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1985.524419] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1985.541076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1985.552884] Lustre: Skipped 1 previous similar message [ 1990.638865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1990.642201] Lustre: Skipped 1 previous similar message [ 1995.748362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1995.757139] Lustre: Skipped 1 previous similar message [ 1999.327298] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1999.516282] Lustre: server umount lustre-OST0000 complete [ 2004.966169] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2004.980078] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2005.017451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2010.663845] Lustre: server umount lustre-MDT0000 complete [ 2015.109395] LustreError: 57578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442990 with bad export cookie 1351509449965544003 [ 2015.120316] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2015.131732] LustreError: 57578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2015.586473] Lustre: server umount lustre-MDT0001 complete [ 2027.916391] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 2031.299395] Key type lgssc unregistered [ 2031.679604] LNet: 61921:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2031.692985] LNetError: 61921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2031.723042] LNet: Removed LNI 192.168.204.140@tcp [ 2032.898729] Key type .llcrypt unregistered [ 2032.902749] Key type ._llcrypt unregistered [ 2083.260504] Key type ._llcrypt registered [ 2083.264735] Key type .llcrypt registered [ 2083.383428] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 2084.509734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2084.656918] alg: No test for adler32 (adler32-zlib) [ 2085.894226] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2086.209111] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 2087.943178] Key type lgssc registered [ 2089.088401] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2093.086079] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2093.103742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2094.459358] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2094.500652] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2094.744438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2095.699828] Lustre: Failing over lustre-MDT0000 [ 2096.066468] Lustre: server umount lustre-MDT0000 complete [ 2105.246964] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 09:44:39 (1788443079) [ 2113.243910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2113.909102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2120.065545] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2131.171345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2131.317761] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2131.386882] Lustre: 64721:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2131.837618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2137.494477] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2146.356157] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2153.511881] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2162.722086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2163.074658] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2163.500580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2170.985725] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2172.540712] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2180.759531] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2188.590844] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2198.509530] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2198.514700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2198.540634] Lustre: Skipped 2 previous similar messages [ 2198.540968] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2198.565807] Lustre: Skipped 2 previous similar messages [ 2203.617536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2203.629506] Lustre: Skipped 2 previous similar messages [ 2204.789771] Lustre: server umount lustre-MDT0000 complete [ 2209.732213] LustreError: 65457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443185 with bad export cookie 7427363340051857489 [ 2209.743105] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2209.753913] LustreError: 65457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2210.267796] Lustre: server umount lustre-MDT0001 complete [ 2215.152698] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2221.713897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2230.242315] Lustre: 63641:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788443189/real 1788443189] req@00000000ac497860 x1875318471175424/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788443205 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2230.283299] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2231.329494] Lustre: 63641:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788443190/real 1788443190] req@0000000095aa37f3 x1875318471175552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788443206 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2232.365636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2234.741322] LustreError: 66573:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2234.768752] LustreError: 66573:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2234.846613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2234.958230] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2235.807358] Lustre: 63644:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788443195/real 1788443195] req@00000000d6be1196 x1875318471175936/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788443211 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2235.875808] LustreError: 66574:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2238.952847] LustreError: 66573:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2238.953523] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2240.455548] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2241.055118] Lustre: 63642:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788443200/real 1788443200] req@00000000aa4ce417 x1875318471176064/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788443216 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2241.091463] Lustre: 63642:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2245.092975] LustreError: 68735:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2245.115012] LustreError: 68735:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2246.111652] Lustre: 63642:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788443205/real 1788443205] req@00000000e0da6625 x1875318471176192/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788443221 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2250.224293] LustreError: 68497:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2250.243068] LustreError: 68497:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2250.969409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2251.603185] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2256.841156] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2256.884267] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2266.156949] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2272.329801] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2282.466838] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2282.491692] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2282.494391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2282.540155] Lustre: Skipped 2 previous similar messages [ 2287.601841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2287.616450] Lustre: Skipped 4 previous similar messages [ 2292.707181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2292.721923] Lustre: Skipped 2 previous similar messages [ 2295.263652] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2295.479462] Lustre: server umount lustre-MDT0000 complete [ 2297.827841] LustreError: 68735:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2297.866313] LustreError: 68735:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2299.821190] LustreError: 65457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443275 with bad export cookie 7427363340051859764 [ 2299.827459] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2299.843702] LustreError: 65457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2300.158387] Lustre: server umount lustre-MDT0001 complete [ 2307.585259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2317.897990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2319.328214] Lustre: 63641:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788443278/real 1788443278] req@0000000003c11209 x1875318471205888/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788443294 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2319.394752] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2324.798728] LustreError: 66573:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2324.853328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2324.903542] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2327.951825] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2331.184477] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2340.993966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2347.013908] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2348.522615] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2358.358622] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2365.768639] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2372.580990] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2372.594076] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2372.603750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2372.611742] Lustre: Skipped 1 previous similar message [ 2372.620691] Lustre: Skipped 2 previous similar messages [ 2374.212167] Lustre: server umount lustre-MDT0000 complete [ 2377.707830] LustreError: 71180:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2377.735788] LustreError: 71180:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 2378.308445] LustreError: 64713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443353 with bad export cookie 7427363340051861143 [ 2378.311306] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2378.316705] LustreError: 64713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2378.611940] Lustre: server umount lustre-MDT0001 complete [ 2386.109522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2389.640786] Lustre: DEBUG MARKER: files compared the same [ 2402.509530] Lustre: server umount lustre-OST0000 complete [ 2421.893607] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 2425.383050] Key type lgssc unregistered [ 2425.706843] LNet: 74461:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2425.712809] LNetError: 74461:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2425.728755] LNet: Removed LNI 192.168.204.140@tcp [ 2426.578870] Key type .llcrypt unregistered [ 2426.581196] Key type ._llcrypt unregistered [ 2455.328286] Key type ._llcrypt registered [ 2455.331063] Key type .llcrypt registered [ 2455.471384] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 2456.811687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2456.903322] alg: No test for adler32 (adler32-zlib) [ 2458.051159] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2458.490741] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 2460.255139] Key type lgssc registered [ 2461.999604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2472.494252] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2472.518516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2474.091448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2480.129535] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug +malloc all [ 2483.682354] LustreError: 75883:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2483.700288] LustreError: 75883:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2488.807528] LustreError: 75882:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2489.723876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2489.971063] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2494.931674] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug +malloc all [ 2503.851314] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2510.489304] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2519.037570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2519.289743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2526.569602] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug +malloc all [ 2535.931662] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2543.859319] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2559.456469] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2559.469045] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2559.478076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2559.498434] Lustre: Skipped 1 previous similar message [ 2562.241493] Lustre: server umount lustre-OST0000 complete [ 2569.703149] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2569.715387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2569.739116] Lustre: Skipped 1 previous similar message [ 2569.764792] Lustre: Skipped 2 previous similar messages [ 2573.330237] Lustre: server umount lustre-MDT0000 complete [ 2574.816701] LustreError: 75883:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2574.836207] LustreError: 75883:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2577.501965] LustreError: 75863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443553 with bad export cookie 14845664168181234244 [ 2577.526824] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2577.846745] Lustre: server umount lustre-MDT0001 complete [ 2589.388249] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 2592.879556] Key type lgssc unregistered [ 2593.241548] LNet: 79443:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2593.260154] LNetError: 79443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2593.284262] LNet: Removed LNI 192.168.204.140@tcp [ 2594.152826] Key type .llcrypt unregistered [ 2594.157537] Key type ._llcrypt unregistered [ 2619.762762] Key type ._llcrypt registered [ 2619.771941] Key type .llcrypt registered [ 2619.893317] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 2620.886813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2620.910818] alg: No test for adler32 (adler32-zlib) [ 2621.920892] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2622.198385] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 2623.847833] Key type lgssc registered [ 2625.069423] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2633.967523] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2634.007066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2649.631396] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2654.751321] LustreError: 80835:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.140@tcp: failed processing log, type 4: rc = -110 [ 2681.504550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2687.934585] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2695.675738] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2704.033044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2704.269251] Lustre: *** cfs_fail_loc=706, val=0*** [ 2704.746648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2704.891164] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2709.415512] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2710.004058] LustreError: 80861:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2710.017376] LustreError: 80861:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2715.113641] LustreError: 80860:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2716.149386] LustreError: 80861:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2716.188121] LustreError: 80861:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2719.877863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2720.159297] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2726.227876] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2736.553583] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2742.934231] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2745.826754] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2745.834340] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2745.851987] LustreError: Skipped 1 previous similar message [ 2745.884098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2745.904803] Lustre: Skipped 1 previous similar message [ 2750.951923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2750.975607] Lustre: Skipped 1 previous similar message [ 2751.630936] Lustre: server umount lustre-OST0000 complete [ 2761.191038] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2761.196980] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2761.207885] Lustre: Skipped 1 previous similar message [ 2761.212561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2761.253676] Lustre: Skipped 1 previous similar message [ 2766.306701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2766.313694] Lustre: Skipped 1 previous similar message [ 2770.404742] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2770.611900] Lustre: server umount lustre-MDT0000 complete [ 2771.427119] LustreError: 83167:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2771.460823] LustreError: 83167:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2774.896131] LustreError: 80843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443750 with bad export cookie 15971365487323968109 [ 2774.904127] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2774.922903] LustreError: 80843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2775.203489] Lustre: server umount lustre-MDT0001 complete [ 2787.792360] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 2790.686230] Key type lgssc unregistered [ 2791.037539] LNet: 84456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2791.055877] LNetError: 84456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2791.095842] LNet: Removed LNI 192.168.204.140@tcp [ 2791.963320] Key type .llcrypt unregistered [ 2791.966096] Key type ._llcrypt unregistered [ 2821.038970] Key type ._llcrypt registered [ 2821.041908] Key type .llcrypt registered [ 2821.161761] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 2822.216030] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2822.234095] alg: No test for adler32 (adler32-zlib) [ 2823.220508] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2823.490988] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 2825.239729] Key type lgssc registered [ 2826.458869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2835.075664] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2835.101051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2841.131874] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2849.780468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2850.471386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2855.887559] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2855.908217] LustreError: 86560:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2855.946297] LustreError: 86560:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2860.001260] LustreError: 86563:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2860.011630] LustreError: 86563:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2865.121263] LustreError: 86564:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2865.154878] LustreError: 86564:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2865.998341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2866.368941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2871.782944] LustreError: 86560:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2871.820561] LustreError: 86560:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2874.122870] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2879.969094] LustreError: 86564:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2879.991194] LustreError: 86564:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2883.561396] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2883.579853] Lustre: MGS: Client c99ebf0e-04ab-4632-8408-37af4cd709dd (at 0@lo) reconnecting [ 2883.598048] LustreError: 86574:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2883.899589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2883.983798] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2889.617799] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2904.551863] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2904.558894] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2904.627915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2909.674468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2909.677364] Lustre: Skipped 1 previous similar message [ 2909.862369] Lustre: server umount lustre-OST0000 complete [ 2914.784346] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2914.795278] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2914.801761] LustreError: Skipped 1 previous similar message [ 2914.807182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2914.826612] Lustre: Skipped 2 previous similar messages [ 2914.863730] Lustre: Skipped 2 previous similar messages [ 2919.905502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2919.924296] Lustre: Skipped 1 previous similar message [ 2920.252571] Lustre: server umount lustre-MDT0000 complete [ 2925.340275] Lustre: server umount lustre-MDT0001 complete [ 2930.281439] Lustre: server umount MGS complete [ 2945.485456] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 09:58:39 (1788443919) [ 2979.947842] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 2992.048943] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 3034.294983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3039.723297] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3050.751873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3051.093277] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3051.166812] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3051.298507] Lustre: lustre-MDT0001: new disk, initializing [ 3051.404057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3051.469165] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3051.477105] Lustre: 93735:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3051.484059] LustreError: 93735:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3055.824824] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3067.266634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3067.558707] Lustre: lustre-OST0000: new disk, initializing [ 3067.561783] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3067.570993] Lustre: 94500:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3067.634269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3075.162803] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3078.304557] Lustre: 94918:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000327528ad x1875319245021824/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788444069 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3086.191771] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3093.678758] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3093.701481] Lustre: MGS: Client 027ab7b5-298e-4ef9-9692-7b314214c16c (at 0@lo) reconnecting [ 3093.774700] Lustre: 95559:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3093.813310] LustreError: 93721:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3093.841749] LustreError: 93721:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 3093.875510] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3094.116469] Lustre: lustre-MDT0000: new disk, initializing [ 3094.367490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3094.429325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3094.459433] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3095.523744] Lustre: 95584:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000004ff5553d x1875319245028736/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788444087 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3100.717345] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3101.388651] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3104.622495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3104.646407] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3104.860096] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3129.320307] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3129.329402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3129.346027] Lustre: Skipped 1 previous similar message [ 3129.358509] Lustre: Skipped 1 previous similar message [ 3135.368373] Lustre: server umount lustre-OST0000 complete [ 3139.558615] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3139.561841] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3139.562847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3139.562852] Lustre: Skipped 2 previous similar messages [ 3139.594968] Lustre: Skipped 1 previous similar message [ 3145.682749] Lustre: server umount lustre-MDT0000 complete [ 3149.802847] LustreError: 96011:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3149.826400] LustreError: 96011:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3156.603282] Lustre: server umount lustre-MDT0001 complete [ 3160.878216] Lustre: server umount MGS complete [ 3171.854629] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 10:02:26 (1788444146) [ 3191.641297] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 3194.511954] Key type lgssc unregistered [ 3194.895496] LNet: 98089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3194.899216] LNetError: 98089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3194.912630] LNet: Removed LNI 192.168.204.140@tcp [ 3195.795668] Key type .llcrypt unregistered [ 3195.797729] Key type ._llcrypt unregistered [ 3214.319110] Key type ._llcrypt registered [ 3214.322649] Key type .llcrypt registered [ 3214.485482] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 3215.901477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3215.966201] alg: No test for adler32 (adler32-zlib) [ 3217.108616] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3217.277932] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 3218.927375] Key type lgssc registered [ 3220.356938] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3230.391430] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3230.435162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3231.920797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3237.194630] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3241.464720] LustreError: 99323:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3241.477577] LustreError: 99323:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3245.503297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3245.838468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3250.706676] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3259.043466] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3265.878417] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3273.798076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3274.201871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3277.304880] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3282.234911] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3291.882927] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3302.881188] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3302.896889] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3302.914557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3305.955542] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3305.958947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3305.964526] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3305.993923] Lustre: Skipped 1 previous similar message [ 3308.499379] Lustre: server umount lustre-OST0000 complete [ 3312.613954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3312.625325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3312.642518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3316.198890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3316.202915] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3318.925798] Lustre: server umount lustre-MDT0000 complete [ 3321.312562] LustreError: 99323:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3321.319576] LustreError: 99323:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3322.866514] LustreError: 102103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444298 with bad export cookie 4246027745053571379 [ 3322.876417] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3322.879484] LustreError: 102103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3323.298299] Lustre: server umount lustre-MDT0001 complete [ 3336.191450] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 3339.532228] Key type lgssc unregistered [ 3339.835732] LNet: 102858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3339.844467] LNetError: 102858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3339.864240] LNet: Removed LNI 192.168.204.140@tcp [ 3341.235062] Key type .llcrypt unregistered [ 3341.248666] Key type ._llcrypt unregistered [ 3360.728494] Key type ._llcrypt registered [ 3360.731497] Key type .llcrypt registered [ 3360.890290] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 3362.152328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3362.186381] alg: No test for adler32 (adler32-zlib) [ 3363.438296] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3363.857620] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 3365.615212] Key type lgssc registered [ 3367.013239] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3373.687751] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3373.701931] LustreError: 103940:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3373.718913] LustreError: 103943:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3373.724796] LustreError: 103940:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4987 [ 3373.730800] LustreError: 103940:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3373.739153] LustreError: 103940:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3373.756721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3375.374796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3383.442556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3384.134509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3390.247705] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3399.011481] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing lsmod [ 3403.183627] LustreError: 105143:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3403.221987] LustreError: 105153:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3403.227865] LustreError: 105143:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4970 [ 3403.236702] LustreError: 105143:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3403.243950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3403.245463] LustreError: 105143:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3403.728251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3408.867141] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3408.874955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3408.895310] Lustre: Skipped 1 previous similar message [ 3408.917459] Lustre: Skipped 2 previous similar messages [ 3409.957596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3409.966210] Lustre: Skipped 2 previous similar messages [ 3414.252618] Lustre: server umount lustre-MDT0000 complete [ 3415.011892] LustreError: 103977:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3415.021738] LustreError: 103977:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3418.382540] LustreError: 103960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444394 with bad export cookie 10868066989481278490 [ 3418.389473] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3418.391619] LustreError: 103960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3420.132481] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3420.133555] LustreError: 103975:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3420.143758] Lustre: Skipped 1 previous similar message [ 3420.145319] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3420.255375] LustreError: 104412:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000056d3bb9a x1875319811233792/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3420.375460] Lustre: server umount lustre-MDT0001 complete [ 3424.692125] Lustre: server umount lustre-OST0000 complete [ 3434.843812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3435.563071] LustreError: 106215:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3435.672022] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3440.918088] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3441.128948] LustreError: 106216:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3446.257560] LustreError: 106215:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3452.051207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3452.354526] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3457.289850] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3466.898454] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3473.743404] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3481.634782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3482.162095] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3488.939354] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3496.661691] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3505.731045] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3516.896978] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3516.904450] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3516.921931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3518.608225] Lustre: server umount lustre-OST0000 complete [ 3524.066385] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3524.072412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3524.098770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3536.357256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3536.361405] Lustre: Skipped 4 previous similar messages [ 3538.399930] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3538.860735] Lustre: server umount lustre-MDT0000 complete [ 3541.475575] LustreError: 106216:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3541.513460] LustreError: 106216:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3543.167398] LustreError: 109021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444518 with bad export cookie 10868066989481279190 [ 3543.174459] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3543.184704] LustreError: 109021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3543.618100] Lustre: server umount lustre-MDT0001 complete [ 3557.090774] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 3560.896361] Key type lgssc unregistered [ 3561.288758] LNet: 109778:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3561.297044] LNetError: 109778:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3561.314508] LNet: Removed LNI 192.168.204.140@tcp [ 3562.285418] Key type .llcrypt unregistered [ 3562.290175] Key type ._llcrypt unregistered [ 3590.775966] Key type ._llcrypt registered [ 3590.778889] Key type .llcrypt registered [ 3590.939285] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 3592.158677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3592.184748] alg: No test for adler32 (adler32-zlib) [ 3593.435176] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3593.816378] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 3595.479257] Key type lgssc registered [ 3596.756758] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3605.668639] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3605.687511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3607.224742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3612.364540] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3616.803016] LustreError: 111198:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3616.840356] LustreError: 111198:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3621.858479] LustreError: 111197:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3622.135848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3622.510856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3627.843214] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3636.825436] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3643.333199] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3650.827611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3651.318517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3652.376757] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3659.283679] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3668.270034] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3675.316262] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3679.380834] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3688.428398] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3688.429860] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3688.469351] LustreError: Skipped 1 previous similar message [ 3688.470389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3688.515443] Lustre: Skipped 1 previous similar message [ 3690.300420] Lustre: server umount lustre-OST0000 complete [ 3698.663312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3698.699845] Lustre: Skipped 1 previous similar message [ 3698.710314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3698.723311] Lustre: Skipped 1 previous similar message [ 3701.670987] Lustre: server umount lustre-MDT0000 complete [ 3703.778106] LustreError: 111197:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3703.831590] LustreError: 111197:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3707.362582] LustreError: 111177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444682 with bad export cookie 3009008521382233103 [ 3707.372957] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3707.375647] LustreError: 111177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3707.864220] Lustre: server umount lustre-MDT0001 complete [ 3722.427820] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 3725.251965] Key type lgssc unregistered [ 3725.620847] LNet: 114842:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3725.629060] LNetError: 114842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3725.641814] LNet: Removed LNI 192.168.204.140@tcp [ 3726.536436] Key type .llcrypt unregistered [ 3726.537791] Key type ._llcrypt unregistered [ 3745.999684] Key type ._llcrypt registered [ 3746.005850] Key type .llcrypt registered [ 3746.167245] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 3747.263365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3747.429041] alg: No test for adler32 (adler32-zlib) [ 3748.495302] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3748.904309] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 3750.591238] Key type lgssc registered [ 3751.775805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3759.572892] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3759.589648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3761.112896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3766.033105] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3771.364786] LustreError: 116066:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3771.397230] LustreError: 116066:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3775.796939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3776.170923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3781.715122] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3791.129428] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3797.374838] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3805.784832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3806.177284] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 3806.365713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3811.854841] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3814.066406] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3822.416993] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3832.295031] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3832.304686] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3832.327314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3836.387678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3836.390426] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3836.407830] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3837.757866] Lustre: server umount lustre-OST0000 complete [ 3842.535535] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3842.557108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3842.586853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3842.597630] Lustre: Skipped 1 previous similar message [ 3846.627797] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3846.628982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3846.669402] Lustre: Skipped 1 previous similar message [ 3848.972644] Lustre: server umount lustre-MDT0000 complete [ 3851.757913] LustreError: 116065:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3851.814389] LustreError: 116065:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3853.679264] LustreError: 116046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444829 with bad export cookie 8455620904030532941 [ 3853.693375] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3853.697107] LustreError: 116046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3854.252466] Lustre: server umount lustre-MDT0001 complete [ 3871.341100] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 3874.761421] Key type lgssc unregistered [ 3875.050066] LNet: 119604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3875.068618] LNetError: 119604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3875.085411] LNet: Removed LNI 192.168.204.140@tcp [ 3876.214633] Key type .llcrypt unregistered [ 3876.217705] Key type ._llcrypt unregistered [ 3906.166156] Key type ._llcrypt registered [ 3906.170893] Key type .llcrypt registered [ 3906.273947] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 3907.818817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3907.873775] alg: No test for adler32 (adler32-zlib) [ 3908.858932] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3909.122040] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 3910.792201] Key type lgssc registered [ 3912.092268] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3920.444131] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3920.480048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3922.015135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3927.811109] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3931.628596] LustreError: 121023:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3931.644412] LustreError: 121023:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3936.742552] LustreError: 121024:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3938.619344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3939.222594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3944.576512] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3954.124951] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3961.949771] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3971.333634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3972.036147] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3979.569475] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3989.119171] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3991.484062] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 4002.284184] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 4007.358333] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 4007.375744] Lustre: Skipped 2 previous similar messages [ 4008.642922] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 4008.750245] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 4012.380592] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 4012.387843] Lustre: Skipped 2 previous similar messages [ 4018.760449] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4018.767328] Lustre: Skipped 1 previous similar message [ 4025.254482] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.40@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 4025.261309] Lustre: Skipped 2 previous similar messages [ 4033.557678] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.40@tcp 192.168.0.[2,10]@tcp [ 4039.995139] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.40/24@tcp in log lustre-MDT0000 [ 4040.003200] Lustre: Skipped 2 previous similar messages [ 4041.409957] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.40/24@tcp [ 4041.418234] Lustre: Skipped 1 previous similar message [ 4061.152258] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4061.161673] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4061.185487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4063.201724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4063.203959] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4063.232774] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4065.568685] Lustre: server umount lustre-OST0000 complete [ 4072.416628] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4072.423338] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4072.434673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4072.439421] Lustre: Skipped 1 previous similar message [ 4078.562347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4078.574052] Lustre: Skipped 3 previous similar messages [ 4083.680952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4083.690032] Lustre: Skipped 1 previous similar message [ 4084.191875] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4084.667689] Lustre: server umount lustre-MDT0000 complete [ 4087.783336] LustreError: 123321:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4087.817950] LustreError: 123321:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4089.648039] LustreError: 123656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445065 with bad export cookie 12335329338391967615 [ 4089.655247] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4089.665119] LustreError: 123656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4090.163788] Lustre: server umount lustre-MDT0001 complete [ 4102.996948] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 4106.740267] Key type lgssc unregistered [ 4107.057390] LNet: 125949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4107.065648] LNetError: 125949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4108.142443] LNet: Removed LNI 192.168.204.140@tcp [ 4109.273076] Key type .llcrypt unregistered [ 4109.279484] Key type ._llcrypt unregistered [ 4141.659771] Key type ._llcrypt registered [ 4141.662330] Key type .llcrypt registered [ 4141.862677] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 4142.857106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4142.893337] alg: No test for adler32 (adler32-zlib) [ 4144.179179] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 4144.491846] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 4146.207488] Key type lgssc registered [ 4147.673752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4156.801269] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4156.828888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4158.372836] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4158.398778] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.40/24@tcp [ 4158.413765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4163.725958] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4167.652657] LustreError: 127515:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4167.672226] LustreError: 127515:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4172.792537] LustreError: 127516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4173.869154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4174.287777] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4174.326505] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.40/24@tcp [ 4174.438899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4179.778972] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4188.863348] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4196.315572] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4204.099527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4204.359444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4210.251659] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4218.910712] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4227.916097] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 4243.430372] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4243.430733] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4243.438759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4243.452785] Lustre: Skipped 1 previous similar message [ 4243.463970] Lustre: Skipped 1 previous similar message [ 4243.919171] Lustre: server umount lustre-OST0000 complete [ 4251.114509] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4251.127989] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4251.135054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4253.672761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4253.688318] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4254.896541] Lustre: server umount lustre-MDT0000 complete [ 4258.786141] LustreError: 129927:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4258.807711] LustreError: 129927:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4259.937188] LustreError: 128256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445235 with bad export cookie 7236426676435903355 [ 4259.945958] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4259.956024] LustreError: 128256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4260.378856] Lustre: server umount lustre-MDT0001 complete [ 4271.581558] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 4275.035768] Key type lgssc unregistered [ 4275.453589] LNet: 131133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4275.476311] LNetError: 131133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4275.490889] LNet: Removed LNI 192.168.204.140@tcp [ 4276.435605] Key type .llcrypt unregistered [ 4276.440198] Key type ._llcrypt unregistered [ 4305.279104] Key type ._llcrypt registered [ 4305.285275] Key type .llcrypt registered [ 4305.440837] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 4321.146753] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 4321.882359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4321.968737] alg: No test for adler32 (adler32-zlib) [ 4323.110681] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 4323.434825] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 4325.119834] Key type lgssc registered [ 4326.579588] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4367.802891] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4367.852622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4369.170794] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4369.225869] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4369.325558] Lustre: lustre-MDT0000: new disk, initializing [ 4369.401165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4369.417957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4374.093569] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4385.631248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4385.789566] Lustre: 134553:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4385.865820] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4385.870607] Lustre: Skipped 1 previous similar message [ 4386.042615] Lustre: lustre-MDT0001: new disk, initializing [ 4386.133944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4386.177561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4386.190585] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4391.113604] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4402.211579] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4409.702358] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4419.642575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4419.994665] Lustre: lustre-OST0000: new disk, initializing [ 4420.000805] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4420.020210] Lustre: 136523:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4420.135123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4428.816730] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4429.396174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4429.413992] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4429.595694] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4440.001164] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4446.538982] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4446.746960] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4451.455635] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4451.591925] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4455.397553] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4455.404061] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4455.406524] LustreError: Skipped 1 previous similar message [ 4455.408592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4455.472442] Lustre: Skipped 1 previous similar message [ 4459.814634] Lustre: server umount lustre-OST0000 complete [ 4465.634943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4465.640756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4465.641922] Lustre: Skipped 1 previous similar message [ 4465.646851] Lustre: Skipped 1 previous similar message [ 4470.621353] Lustre: server umount lustre-MDT0000 complete [ 4475.877566] LustreError: 134564:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4475.886952] LustreError: 134564:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4476.041279] LustreError: 135357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445451 with bad export cookie 6214335398433833658 [ 4476.051324] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4476.056309] LustreError: 135357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4476.387765] Lustre: server umount lustre-MDT0001 complete [ 4489.760261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4490.352246] LustreError: 138740:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4490.477168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4495.840745] LustreError: 138741:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4496.447390] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4500.977395] LustreError: 138740:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4506.087497] LustreError: 138741:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4507.153392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4507.595253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4513.033321] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4522.590833] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4530.080904] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4538.243895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4538.570839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4545.641913] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4554.530956] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4560.053953] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4561.408637] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 4567.438800] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4567.791969] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4577.683879] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4577.980906] Lustre: lustre-OST0001: new disk, initializing [ 4577.988967] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4577.999205] Lustre: 142119:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4578.124948] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4585.930513] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4586.627877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4586.642622] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4586.781131] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4596.222372] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4596.453313] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4602.724982] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4603.031956] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4609.494249] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8dd447f64800.ost_server_uuid 50 [ 4611.241791] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8dd447f64800.ost_server_uuid in IDLE state after 0 sec [ 4622.824219] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4622.833046] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4622.835600] LustreError: Skipped 1 previous similar message [ 4622.841767] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4622.856284] Lustre: Skipped 1 previous similar message [ 4622.887962] Lustre: Skipped 1 previous similar message [ 4626.409283] Lustre: server umount lustre-OST0001 complete [ 4627.949811] LustreError: 142129:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4627.982561] LustreError: 142129:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4633.077835] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4633.084751] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4633.090475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4633.124421] Lustre: Skipped 1 previous similar message [ 4633.141243] Lustre: Skipped 1 previous similar message [ 4636.777219] Lustre: server umount lustre-OST0000 complete [ 4643.302395] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4643.305193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4643.323036] Lustre: Skipped 1 previous similar message [ 4643.340661] Lustre: Skipped 1 previous similar message [ 4647.475119] Lustre: server umount lustre-MDT0000 complete [ 4648.416244] LustreError: 139468:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4648.437648] LustreError: 139468:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4651.923287] LustreError: 138720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445627 with bad export cookie 6214335398433834547 [ 4651.925351] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4651.935815] LustreError: 138720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4652.691861] Lustre: server umount lustre-MDT0001 complete [ 4676.926629] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 4680.134693] Key type lgssc unregistered [ 4680.518489] LNet: 145129:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4680.522201] LNetError: 145129:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4680.533740] LNet: Removed LNI 192.168.204.140@tcp [ 4681.323835] Key type .llcrypt unregistered [ 4681.325494] Key type ._llcrypt unregistered [ 4729.914715] Key type ._llcrypt registered [ 4729.916346] Key type .llcrypt registered [ 4730.024884] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 4746.958737] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 4748.484013] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4748.539567] alg: No test for adler32 (adler32-zlib) [ 4749.713074] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 4749.986102] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 4751.687882] Key type lgssc registered [ 4753.142803] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4795.101445] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4795.132846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4796.449160] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4796.488571] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4796.574535] Lustre: lustre-MDT0000: new disk, initializing [ 4796.651250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4796.666594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4801.537355] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4812.954746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4813.195683] Lustre: 149288:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4813.267733] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4813.273971] Lustre: Skipped 1 previous similar message [ 4813.370413] Lustre: lustre-MDT0001: new disk, initializing [ 4813.494839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4813.550616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4813.568925] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4818.790053] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4829.980379] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4836.385756] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4844.060326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4844.526107] Lustre: lustre-OST0000: new disk, initializing [ 4844.530241] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4844.547269] Lustre: 151256:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4844.671437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4850.267394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4850.286410] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4850.421932] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4852.446043] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4863.573896] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4869.998064] Lustre: Failing over lustre-OST0000 [ 4870.353961] Lustre: server umount lustre-OST0000 complete [ 4886.495264] Lustre: 147513:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445846/real 1788445846] req@000000005da1b557 x1875321264573568/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788445862 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4886.508441] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4886.531749] Lustre: 147513:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4886.582894] Lustre: Skipped 1 previous similar message [ 4887.519119] Lustre: 147511:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445846/real 1788445846] req@000000007157f47d x1875321264572928/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788445862 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4887.569937] Lustre: 147511:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4890.119496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4890.558946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4890.574387] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4891.556116] Lustre: 147514:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445851/real 1788445851] req@0000000082165189 x1875321264574592/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788445867 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4896.686815] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4897.739218] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4897.760077] Lustre: 147512:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445857/real 1788445857] req@000000004d59b59f x1875321264575744/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788445873 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4897.778398] Lustre: 147512:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4898.428705] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4898.428790] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4901.708401] Lustre: Failing over lustre-MDT0000 [ 4901.857697] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4901.942890] Lustre: server umount lustre-MDT0000 complete [ 4906.977132] LustreError: 149297:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4907.001742] LustreError: 149297:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4908.642083] LustreError: 150510:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4912.098517] LustreError: 149297:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4912.113937] LustreError: 149297:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4917.221263] LustreError: 149300:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4917.242605] LustreError: 149300:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4921.518598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4921.698971] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4922.085742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4922.170059] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4924.005022] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4927.477501] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4927.494318] Lustre: Skipped 1 previous similar message [ 4927.524036] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4927.588620] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4927.866418] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4937.699667] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4937.700987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4937.709527] Lustre: Skipped 3 previous similar messages [ 4942.197276] Lustre: server umount lustre-OST0000 complete [ 4947.936786] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4947.941281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4947.964908] Lustre: Skipped 1 previous similar message [ 4947.987692] Lustre: Skipped 2 previous similar messages [ 4952.555034] Lustre: server umount lustre-MDT0000 complete [ 4953.059554] LustreError: 149301:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4953.112952] LustreError: 149301:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4956.875953] LustreError: 149281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445932 with bad export cookie 15392488744420417482 [ 4956.878803] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4956.882048] LustreError: 149281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4957.124116] Lustre: server umount lustre-MDT0001 complete [ 4972.657972] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 4977.413789] Key type lgssc unregistered [ 4977.718925] LNet: 154926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4977.766347] LNetError: 154926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4978.791166] LNet: Removed LNI 192.168.204.140@tcp [ 4979.884848] Key type .llcrypt unregistered [ 4979.887390] Key type ._llcrypt unregistered [ 5018.048249] Key type ._llcrypt registered [ 5018.051413] Key type .llcrypt registered [ 5018.189371] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 5019.409503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5019.536093] alg: No test for adler32 (adler32-zlib) [ 5020.655420] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5020.922967] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 5022.647166] Key type lgssc registered [ 5023.946720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5032.035109] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5032.051308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5033.486424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5038.204850] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5043.692095] LustreError: 156449:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5043.713859] LustreError: 156449:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5047.779203] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5048.821360] LustreError: 156448:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5052.252154] LustreError: 156477:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5052.262314] LustreError: 156477:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5052.272169] LustreError: 156477:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 5052.665755] Lustre: server umount lustre-MDT0000 complete [ 5068.945927] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 5082.419981] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 5125.433098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5125.619239] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5125.653095] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5125.708607] Lustre: lustre-MDT0000: new disk, initializing [ 5125.769887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5125.783941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5129.428354] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5139.992522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5140.089206] Lustre: 159780:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5140.141151] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5140.147279] Lustre: Skipped 1 previous similar message [ 5140.239267] Lustre: lustre-MDT0001: new disk, initializing [ 5140.414233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5140.484870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5140.504061] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5146.075507] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5159.158604] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5165.941890] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5172.725826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5173.021724] Lustre: lustre-OST0000: new disk, initializing [ 5173.025829] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5173.041845] Lustre: 161749:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5173.155149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5178.968694] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5183.085936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5183.104806] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5183.197105] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5188.812326] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5193.028405] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5195.437427] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 10:36:09 (1788446169) [ 5468.138331] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5468.147845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5468.167926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5470.176300] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5470.184509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5470.194227] Lustre: Skipped 1 previous similar message [ 5470.205239] Lustre: Skipped 2 previous similar messages [ 5472.761534] Lustre: server umount lustre-MDT0000 complete [ 5475.299277] LustreError: 160581:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5475.310030] LustreError: 160581:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5476.282640] LustreError: 161748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446451 with bad export cookie 876643478358279139 [ 5476.288766] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5476.292589] LustreError: 161748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5480.415794] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5480.416343] LustreError: 159788:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5480.446438] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5482.605675] Lustre: server umount lustre-MDT0001 complete [ 5500.896781] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5501.110531] Lustre: server umount lustre-OST0000 complete [ 5511.789540] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 5521.000537] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 5558.849134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5559.092367] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5559.124448] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5559.245061] Lustre: lustre-MDT0000: new disk, initializing [ 5559.352226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5559.369438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5564.585349] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5576.590983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5576.767640] Lustre: 166416:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5576.887758] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5576.892386] Lustre: Skipped 1 previous similar message [ 5577.029629] Lustre: lustre-MDT0001: new disk, initializing [ 5577.229215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5577.276907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5577.290317] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5581.437194] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5591.734951] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5598.556502] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5605.810352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5606.118873] Lustre: lustre-OST0000: new disk, initializing [ 5606.128751] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5606.139737] Lustre: 168385:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5606.255920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5611.713120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5611.732974] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5611.957382] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5614.512787] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5623.977814] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5628.572631] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5628.748905] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5633.060991] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5633.265626] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5637.609960] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5637.611341] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5637.642757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5637.649754] Lustre: Skipped 2 previous similar messages [ 5642.723369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5642.729320] Lustre: Skipped 1 previous similar message [ 5649.375200] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5649.545538] Lustre: server umount lustre-OST0000 complete [ 5653.984428] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5653.992114] LustreError: Skipped 1 previous similar message [ 5653.997307] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5654.009016] Lustre: Skipped 1 previous similar message [ 5654.023179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5654.030512] Lustre: Skipped 2 previous similar messages [ 5659.535169] Lustre: server umount lustre-MDT0000 complete [ 5663.207929] LustreError: 166429:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5663.226708] LustreError: 166429:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5663.400225] LustreError: 166409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446639 with bad export cookie 876643478358437969 [ 5663.410926] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5663.414339] LustreError: 166409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5663.767508] Lustre: server umount lustre-MDT0001 complete [ 5672.016853] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:44:06 (1788446646) [ 5687.835098] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 5697.278449] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 5736.506912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5736.840346] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5736.866980] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5736.953159] Lustre: lustre-MDT0000: new disk, initializing [ 5737.019129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5737.032704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5742.399541] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5753.093506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5753.232375] Lustre: 173143:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5753.560148] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5758.105772] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5767.511556] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5773.791942] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5780.574685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5780.827618] Lustre: lustre-OST0000: new disk, initializing [ 5780.830827] Lustre: Skipped 1 previous similar message [ 5780.834825] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5780.839780] Lustre: Skipped 2 previous similar messages [ 5780.845216] Lustre: 175120:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5787.192027] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5787.212892] Lustre: Skipped 1 previous similar message [ 5787.221511] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5787.309499] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5787.382307] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5796.042350] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5802.979802] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5802.995515] Lustre: Skipped 1 previous similar message [ 5803.007971] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5803.008742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5803.026380] Lustre: Skipped 3 previous similar messages [ 5817.311774] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5817.571805] Lustre: server umount lustre-OST0000 complete [ 5823.461411] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5823.475990] Lustre: Skipped 2 previous similar messages [ 5827.726835] Lustre: server umount lustre-MDT0000 complete [ 5828.576091] LustreError: 173149:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5828.590206] LustreError: 173149:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5832.120519] LustreError: 173135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446807 with bad export cookie 876643478358438851 [ 5832.124771] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5832.138065] LustreError: 173135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5832.693938] Lustre: server umount lustre-MDT0001 complete [ 5841.274194] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:46:55 (1788446815) [ 5858.102299] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 5867.283679] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 5901.654865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5901.927754] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5901.979624] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5902.071721] Lustre: lustre-MDT0000: new disk, initializing [ 5902.168018] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5902.171945] Lustre: Skipped 2 previous similar messages [ 5902.185604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5906.424824] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5915.923574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5916.064489] Lustre: 179447:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5916.328439] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5920.816757] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5928.676987] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5933.627262] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5939.128084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5939.384599] Lustre: 181422:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5944.953683] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5948.092491] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5954.542874] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5963.071215] Lustre: server umount lustre-OST0000 complete [ 5967.328382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5967.335249] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5967.346491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5967.354693] Lustre: Skipped 6 previous similar messages [ 5977.134727] Lustre: server umount lustre-MDT0000 complete [ 5979.105243] LustreError: 180757:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5979.119512] LustreError: 180757:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5979.616808] Lustre: 155760:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788446939/real 1788446939] req@0000000011e20c4a x1875321549524224/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788446955 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5979.657044] Lustre: 155760:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5981.245252] LustreError: 179438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446956 with bad export cookie 876643478358439929 [ 5981.249548] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5981.263914] LustreError: 179438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5981.694328] Lustre: server umount lustre-MDT0001 complete [ 5994.275564] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 5997.204585] Key type lgssc unregistered [ 5997.565679] LNet: 183281:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5997.571522] LNetError: 183281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5997.601264] LNet: Removed LNI 192.168.204.140@tcp [ 5998.203375] Key type .llcrypt unregistered [ 5998.205210] Key type ._llcrypt unregistered [ 6020.096623] Key type ._llcrypt registered [ 6020.099581] Key type .llcrypt registered [ 6020.187575] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 6021.358117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6021.402923] alg: No test for adler32 (adler32-zlib) [ 6022.471486] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6022.716801] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 6024.375710] Key type lgssc registered [ 6025.591136] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6032.471649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6032.492734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6033.882125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6038.327532] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6044.133448] LustreError: 184698:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6044.144822] LustreError: 184698:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6046.392738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6046.817433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6050.886592] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6057.873632] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6063.567586] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6069.923639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6070.393666] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6076.479753] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6078.576135] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6082.890803] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6102.497407] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6102.512462] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6102.539537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6104.546401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6104.546911] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6104.583029] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6107.532691] Lustre: server umount lustre-OST0000 complete [ 6113.249898] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6113.257449] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6113.268764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6113.271274] Lustre: Skipped 1 previous similar message [ 6117.401731] Lustre: server umount lustre-MDT0000 complete [ 6119.906265] LustreError: 184698:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6119.922060] LustreError: 184698:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6121.163565] LustreError: 184679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447096 with bad export cookie 9938894586871057245 [ 6121.168046] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6121.172581] LustreError: 184679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6121.414254] Lustre: server umount lustre-MDT0001 complete [ 6133.116984] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 6136.332503] Key type lgssc unregistered [ 6136.666914] LNet: 188257:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6136.701335] LNetError: 188257:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6136.735248] LNet: Removed LNI 192.168.204.140@tcp [ 6137.439208] Key type .llcrypt unregistered [ 6137.443517] Key type ._llcrypt unregistered [ 6160.197392] Key type ._llcrypt registered [ 6160.199366] Key type .llcrypt registered [ 6160.305235] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 6161.270755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6161.386138] alg: No test for adler32 (adler32-zlib) [ 6162.423739] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6162.701424] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 6164.423204] Key type lgssc registered [ 6165.592489] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6173.712510] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6173.727326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6175.040150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6179.091466] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6184.421822] LustreError: 189677:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6184.435443] LustreError: 189677:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6187.872819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6188.195342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6193.276408] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6201.784831] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6207.939689] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6214.798340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6215.126671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6220.285767] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6221.477977] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6229.737811] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6240.736498] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6240.743566] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6240.764054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6241.923667] Lustre: lustre-OST0000: Not available for connect from 192.168.204.40@tcp (stopping) [ 6242.794588] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6242.820230] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6245.334229] Lustre: server umount lustre-OST0000 complete [ 6253.536134] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8dd446c61800.ost_server_uuid 50 [ 6255.259035] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6268.386588] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6268.389444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6268.398329] Lustre: Skipped 1 previous similar message [ 6268.413517] Lustre: Skipped 3 previous similar messages [ 6273.504450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6273.509237] Lustre: Skipped 1 previous similar message [ 6274.593980] Lustre: server umount lustre-MDT0000 complete [ 6277.958680] LustreError: 189658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447253 with bad export cookie 4651770709439429084 [ 6277.960183] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6277.969658] LustreError: 189658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6278.304459] Lustre: server umount lustre-MDT0001 complete [ 6286.423946] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 10:54:21 (1788447261) [ 6292.873528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6293.326824] LustreError: 193592:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6293.402563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6297.945922] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6298.593056] LustreError: 193593:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6302.690662] LustreError: 193592:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6306.442541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6306.904914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6311.111469] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6317.984480] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6323.299830] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6329.408387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6329.803723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6335.463284] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6337.966850] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6341.511757] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6347.484864] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6347.642181] Lustre: lustre-OST0001: new disk, initializing [ 6347.650356] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6347.659934] Lustre: 196433:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6347.761205] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6353.723857] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6356.036276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6356.052478] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6356.129503] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6362.707055] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6366.694939] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6366.701337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6366.705430] Lustre: Skipped 1 previous similar message [ 6371.459348] Lustre: server umount lustre-OST0000 complete [ 6371.818372] LustreError: 195415:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6371.835072] LustreError: 195415:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6378.723593] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6379.012779] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6382.057476] LustreError: 195414:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6382.076290] LustreError: 195414:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6383.511480] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6383.754460] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6402.532550] LustreError: 197892:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6402.534700] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6402.539475] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6402.544132] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6402.544141] Lustre: Skipped 2 previous similar messages [ 6402.565335] LustreError: 197892:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6403.921461] Lustre: server umount lustre-OST0001 complete [ 6413.846167] Lustre: server umount lustre-MDT0000 complete [ 6417.400364] LustreError: 193573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447393 with bad export cookie 4651770709439430162 [ 6417.408365] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6417.411157] LustreError: 193573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6417.771464] Lustre: server umount lustre-MDT0001 complete [ 6439.592815] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 10:56:54 (1788447414) [ 6446.003162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6446.322926] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6446.361564] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6446.536364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6451.360753] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6461.620802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6461.740648] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6461.794629] Lustre: 199874:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6467.559364] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6475.946717] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6483.471082] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6490.202915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6490.309343] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6496.856429] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6498.994431] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6504.854509] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6512.914611] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6513.124712] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6513.445769] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6513.453144] Lustre: Skipped 2 previous similar messages [ 6521.093346] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6521.532280] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6529.309661] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6534.624567] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6534.632647] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6534.639112] Lustre: Skipped 3 previous similar messages [ 6534.647143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6534.656202] Lustre: Skipped 5 previous similar messages [ 6538.932284] Lustre: server umount lustre-OST0000 complete [ 6540.369782] LustreError: 201724:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6557.671116] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6557.672140] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6557.682273] LustreError: Skipped 2 previous similar messages [ 6557.721462] Lustre: Skipped 2 previous similar messages [ 6562.439701] Lustre: server umount lustre-OST0001 complete [ 6567.907380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6567.920786] Lustre: Skipped 7 previous similar messages [ 6572.319597] Lustre: server umount lustre-MDT0000 complete [ 6575.321103] LustreError: 199865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447550 with bad export cookie 4651770709439431569 [ 6575.332625] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6575.603206] Lustre: server umount lustre-MDT0001 complete [ 6600.149686] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 10:59:34 (1788447574) [ 6615.175331] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 6624.822729] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 6661.347682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6661.605613] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6661.644173] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6661.745856] Lustre: lustre-MDT0000: new disk, initializing [ 6661.838033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6661.850944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6666.841532] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6679.139468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6679.265977] Lustre: 207971:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6679.355650] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6679.362709] Lustre: Skipped 1 previous similar message [ 6679.463699] Lustre: lustre-MDT0001: new disk, initializing [ 6679.649364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6679.660807] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6684.124335] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6694.253654] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6701.097093] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6710.240634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6710.597606] Lustre: lustre-OST0000: new disk, initializing [ 6710.602214] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6710.614427] Lustre: 209950:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6716.828806] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6718.060116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6718.076348] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6718.228770] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6727.244300] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6731.213420] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6731.443843] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6735.326636] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6735.550740] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6738.912764] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6738.917921] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6738.922911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6738.929831] Lustre: Skipped 3 previous similar messages [ 6738.935561] Lustre: Skipped 1 previous similar message [ 6743.518623] Lustre: server umount lustre-OST0000 complete [ 6753.331439] Lustre: server umount lustre-MDT0000 complete [ 6754.281152] LustreError: 207983:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6754.297131] LustreError: 207983:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 6757.335773] LustreError: 207963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447732 with bad export cookie 4651770709439433354 [ 6757.352113] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6757.357302] LustreError: 207963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6757.822994] Lustre: server umount lustre-MDT0001 complete [ 6766.605886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6772.372848] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6780.591379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6785.850548] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6793.532923] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6799.936439] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6806.080854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6811.677583] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6820.522506] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6826.360988] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6826.731158] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6832.260060] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6832.563889] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6836.712045] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6836.714806] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6836.729411] Lustre: Skipped 3 previous similar messages [ 6840.914424] Lustre: server umount lustre-OST0000 complete [ 6848.646300] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6853.138548] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6882.305312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6888.913515] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6895.609733] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6900.217995] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 6902.130704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6902.151096] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6902.151727] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6905.544774] Lustre: server umount lustre-OST0000 complete [ 6912.488860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6912.496276] Lustre: Skipped 4 previous similar messages [ 6918.697950] Lustre: server umount lustre-MDT0000 complete [ 6922.212558] LustreError: 214015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447897 with bad export cookie 4651770709439434229 [ 6922.231537] LustreError: 214015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6922.232944] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6931.697987] Lustre: server umount lustre-MDT0001 complete [ 6940.872662] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 11:05:15 (1788447915) [ 6947.020957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6947.821946] LustreError: 218140:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6947.850107] LustreError: 218140:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6947.930914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6947.937697] Lustre: Skipped 6 previous similar messages [ 6952.102782] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6961.561887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6966.696285] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6976.413633] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6983.632847] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6991.336936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6998.940794] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7008.704443] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7015.046822] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7015.474793] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7020.676521] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7020.942803] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7028.150371] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7028.514085] Lustre: lustre-OST0001: new disk, initializing [ 7028.517070] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7028.527065] Lustre: 221512:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7035.221809] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7037.551094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7037.561113] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7037.664915] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7044.376434] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7049.836333] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7050.015127] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7055.178450] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7055.440494] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7058.414534] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7058.419632] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7058.429247] LustreError: Skipped 1 previous similar message [ 7058.449021] Lustre: Skipped 3 previous similar messages [ 7063.718678] Lustre: server umount lustre-OST0001 complete [ 7069.763590] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7070.035984] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7074.588472] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7074.794372] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7103.497190] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7109.101641] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7109.113085] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7109.120454] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7109.141580] Lustre: Skipped 2 previous similar messages [ 7110.771923] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7116.931736] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7118.503472] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7124.451630] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7124.464860] LustreError: Skipped 1 previous similar message [ 7126.602207] Lustre: server umount lustre-OST0001 complete [ 7150.503683] Lustre: server umount lustre-OST0000 complete [ 7160.561188] Lustre: server umount lustre-MDT0000 complete [ 7164.517003] LustreError: 218121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448140 with bad export cookie 4651770709439435202 [ 7164.526509] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7164.532678] LustreError: 218121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7171.031412] Lustre: server umount lustre-MDT0001 complete [ 7208.203639] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 11:09:40 (1788448180) [ 7217.071732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7217.506980] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7217.568386] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7221.816312] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7231.319286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7231.551782] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7231.589870] Lustre: 226760:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7236.864593] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7244.538169] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7250.653326] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7257.999667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7258.408188] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7267.354689] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7276.892055] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7288.453447] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7288.521921] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7293.784956] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7300.822704] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7302.902522] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7308.289991] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7308.580457] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7314.451540] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7314.690697] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7322.350133] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8dd44ab6d000.ost_server_uuid 50 [ 7324.369930] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8dd44ab6d000.ost_server_uuid in IDLE state after 0 sec [ 7325.388073] Lustre: Permanently deactivating lustre-OST0001 [ 7325.408155] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7329.508467] Lustre: Permanently reactivating lustre-OST0001 [ 7329.526306] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7329.533273] Lustre: Skipped 2 previous similar messages [ 7338.466930] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7338.473451] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7338.490818] Lustre: Skipped 6 previous similar messages [ 7338.500798] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7338.503849] Lustre: Skipped 15 previous similar messages [ 7341.832668] Lustre: server umount lustre-OST0001 complete [ 7344.102048] LustreError: 228603:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7344.132454] LustreError: 228603:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 36 previous similar messages [ 7351.957670] Lustre: server umount lustre-OST0000 complete [ 7362.423325] Lustre: server umount lustre-MDT0000 complete [ 7366.807088] LustreError: 229634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448342 with bad export cookie 4651770709439436490 [ 7366.814950] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7367.143572] Lustre: server umount lustre-MDT0001 complete [ 7392.894576] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 11:12:47 (1788448367) [ 7400.488933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7400.791342] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7400.827880] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7400.833824] Lustre: Skipped 2 previous similar messages [ 7406.290652] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7416.073377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7416.214156] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7416.273076] Lustre: 233460:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7422.058958] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7431.216672] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7437.985218] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7446.041915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7446.361300] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7446.428218] Lustre: Permanently deactivating lustre-OST0000 [ 7446.445882] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7453.205783] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7453.836677] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7461.326779] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7468.907225] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7468.990838] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7469.247087] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7469.260581] Lustre: Skipped 11 previous similar messages [ 7475.798456] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7476.445792] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7483.553958] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7485.518703] Lustre: Permanently reactivating lustre-OST0000 [ 7485.544707] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7485.556201] Lustre: Skipped 2 previous similar messages [ 7491.641052] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 7491.673596] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 39 seconds. I think it's dead, and I am evicting it. exp 0000000002ca49d2, cur 1788448467 deadline 1788448458 last 1788448428 [ 7491.739542] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7517.153961] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7517.291907] Lustre: server umount lustre-OST0001 complete [ 7527.458354] Lustre: server umount lustre-OST0000 complete [ 7537.828511] Lustre: server umount lustre-MDT0000 complete [ 7542.922146] LustreError: 233450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448518 with bad export cookie 4651770709439438170 [ 7542.931374] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7542.936559] LustreError: 233450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7543.824440] Lustre: server umount lustre-MDT0001 complete [ 7559.781663] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 7563.401784] Key type lgssc unregistered [ 7563.908630] LNet: 238348:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7563.917881] LNetError: 238348:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7563.953168] LNet: Removed LNI 192.168.204.140@tcp [ 7565.491398] Key type .llcrypt unregistered [ 7565.495947] Key type ._llcrypt unregistered [ 7592.769430] Key type ._llcrypt registered [ 7592.772620] Key type .llcrypt registered [ 7592.946528] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 7594.837137] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7594.933533] alg: No test for adler32 (adler32-zlib) [ 7595.986386] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7596.369088] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 7598.119179] Key type lgssc registered [ 7599.775099] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7612.007941] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7612.039122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7613.641076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7619.493110] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7624.169783] LustreError: 239818:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7624.198720] LustreError: 239818:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7629.306702] LustreError: 239819:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7629.628126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7629.821784] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7629.856109] Lustre: 239806:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7629.864578] Lustre: Permanently deactivating lustre-MDT0001 [ 7629.877941] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7630.310147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7632.691471] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7632.708738] LustreError: 239849:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7632.737316] LustreError: 239849:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7632.754357] LustreError: 239849:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 1, failed: rc = -108 [ 7635.749899] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7645.692774] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7652.459935] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7660.134235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7660.765262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7667.473774] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7675.109570] LustreError: 241660:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7675.118818] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7676.580176] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7680.486339] LustreError: 241660:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7683.574121] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7684.167859] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7684.444334] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7689.718963] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7689.720411] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7690.994188] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7699.442756] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7703.706334] Lustre: Permanently reactivating lustre-MDT0001 [ 7703.724977] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7703.729394] Lustre: Skipped 2 previous similar messages [ 7705.223305] Lustre: lustre-MDT0001: Client 6d745b4b-5ae0-48c3-a6e1-f4a95636eea8 (at 192.168.204.40@tcp) reconnecting [ 7713.421726] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7715.194174] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7722.222314] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7722.482367] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7727.787043] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7728.059137] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7730.850508] Lustre: Permanently deactivating lustre-MDT0001 [ 7730.865173] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7730.876258] Lustre: Skipped 2 previous similar messages [ 7737.446791] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7746.020415] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7746.024536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7746.027087] Lustre: Skipped 3 previous similar messages [ 7746.038640] Lustre: Skipped 3 previous similar messages [ 7747.872197] Lustre: server umount lustre-MDT0000 complete [ 7751.137401] LustreError: 240563:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7751.150411] LustreError: 240563:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7751.275085] LustreError: 239797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448726 with bad export cookie 13967986821543481099 [ 7751.277852] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7751.284094] LustreError: 239797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7751.607485] Lustre: server umount lustre-MDT0001 complete [ 7757.493376] Lustre: server umount lustre-OST0000 complete [ 7769.187814] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 11:19:03 (1788448743) [ 7781.000396] Lustre: server umount lustre-OST0001 complete [ 7789.043625] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 7800.360655] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 7839.640865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7839.865807] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7839.879310] Lustre: Skipped 2 previous similar messages [ 7839.911392] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7839.994077] Lustre: lustre-MDT0000: new disk, initializing [ 7840.125786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7840.146182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7845.107714] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7855.619073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7855.737958] Lustre: 248079:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7855.780473] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7855.783455] Lustre: Skipped 1 previous similar message [ 7855.859461] Lustre: lustre-MDT0001: new disk, initializing [ 7855.938392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7855.970728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7855.984844] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7860.411572] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7870.137731] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7876.105674] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7883.055442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7883.230324] Lustre: lustre-OST0000: new disk, initializing [ 7883.233848] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7883.238576] Lustre: 250047:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7883.323038] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7891.193388] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7892.115444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7892.124555] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7892.361574] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7902.573044] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7905.416707] LustreError: 248085:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7914.854368] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7915.000720] Lustre: lustre-OST0001: new disk, initializing [ 7915.005580] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7915.013359] Lustre: 251165:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7915.439555] LustreError: 248085:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7915.492543] LustreError: 250820:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7921.816896] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7922.845945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7922.863237] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7922.990897] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7925.520172] LustreError: 250820:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7934.748539] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7936.970202] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7943.658884] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7943.661323] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7943.681389] LustreError: Skipped 1 previous similar message [ 7943.685763] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7943.725801] Lustre: Skipped 1 previous similar message [ 7946.031139] Lustre: server umount lustre-OST0001 complete [ 7948.774546] LustreError: 250031:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7948.794220] LustreError: 250031:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7964.134710] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7964.135407] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7964.143337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7964.157859] Lustre: Skipped 2 previous similar messages [ 7969.263225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7969.264233] LustreError: 250030:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7969.275677] Lustre: Skipped 1 previous similar message [ 7969.303639] LustreError: 250030:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 7970.508887] Lustre: server umount lustre-OST0000 complete [ 7978.978616] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7978.993555] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7979.013039] Lustre: Skipped 1 previous similar message [ 7979.019747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7989.215205] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7989.444930] Lustre: server umount lustre-MDT0000 complete [ 7993.847335] LustreError: 252148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448969 with bad export cookie 13967986821543483906 [ 7993.850405] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7993.872120] LustreError: 252148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8008.159373] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8008.466748] Lustre: server umount lustre-MDT0001 complete [ 8022.465822] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 8025.846925] Key type lgssc unregistered [ 8026.175549] LNet: 253208:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8026.190583] LNetError: 253208:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8026.223258] LNet: Removed LNI 192.168.204.140@tcp [ 8027.489607] Key type .llcrypt unregistered [ 8027.493099] Key type ._llcrypt unregistered [ 8069.802383] Key type ._llcrypt registered [ 8069.810196] Key type .llcrypt registered [ 8069.927897] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 8071.838645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8071.880208] alg: No test for adler32 (adler32-zlib) [ 8072.992657] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8073.319710] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 8075.047706] Key type lgssc registered [ 8076.096773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8084.457928] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8084.477759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8085.813129] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8085.835870] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8086.027602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8090.687456] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 8099.920746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8100.071972] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8100.125648] Lustre: 255425:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8100.317638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8104.989630] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 8115.226429] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8122.387553] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8130.951456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8131.205973] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8131.383184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8137.977729] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 8141.566125] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8146.718311] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8203.248384] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8203.264101] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8203.296943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8203.309794] Lustre: Skipped 1 previous similar message [ 8207.430582] Lustre: server umount lustre-OST0000 complete [ 8212.306806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8227.104597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8227.630820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8234.576276] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 8243.003123] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8243.024063] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8243.540306] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8244.076616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8244.100346] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8284.129606] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8284.138219] LustreError: Skipped 1 previous similar message [ 8284.141857] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8284.151093] Lustre: Skipped 1 previous similar message [ 8284.157291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8285.153268] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8285.160620] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8285.177038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8285.181713] Lustre: Skipped 1 previous similar message [ 8289.570225] Lustre: server umount lustre-OST0000 complete [ 8294.893945] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8294.899490] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8294.918948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8299.860798] Lustre: server umount lustre-MDT0000 complete [ 8304.256210] LustreError: 255416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449279 with bad export cookie 7800656726787666179 [ 8304.256467] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8304.265251] LustreError: 255416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8304.495139] Lustre: server umount lustre-MDT0001 complete [ 8317.660944] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 8320.777196] Key type lgssc unregistered [ 8321.174780] LNet: 260850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8321.183687] LNetError: 260850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8322.217040] LNet: Removed LNI 192.168.204.140@tcp [ 8323.277919] Key type .llcrypt unregistered [ 8323.286737] Key type ._llcrypt unregistered [ 8349.992383] Key type ._llcrypt registered [ 8349.996130] Key type .llcrypt registered [ 8350.132303] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 8351.390094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8351.447864] alg: No test for adler32 (adler32-zlib) [ 8352.796597] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8353.099675] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 8354.807735] Key type lgssc registered [ 8355.979763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8364.705360] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8364.720026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8366.394882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8371.572870] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 8375.786184] LustreError: 262269:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8375.800474] LustreError: 262269:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8380.818507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8380.898048] LustreError: 262270:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8381.173829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8385.786751] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 8394.605566] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8402.335475] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8409.385704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8409.632427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8410.678144] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8416.733294] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 8424.829868] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8451.558717] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8451.565259] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8451.572925] LustreError: Skipped 1 previous similar message [ 8451.577404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8451.593479] Lustre: Skipped 1 previous similar message [ 8453.978481] Lustre: server umount lustre-OST0000 complete [ 8458.208413] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8458.215579] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8458.235494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8458.241735] Lustre: Skipped 1 previous similar message [ 8461.795682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8461.797939] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8464.073937] Lustre: server umount lustre-MDT0000 complete [ 8466.914973] LustreError: 262997:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8466.937267] LustreError: 262997:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8467.630904] LustreError: 262249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449443 with bad export cookie 14797458675274768117 [ 8467.636216] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8467.648066] LustreError: 262249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8467.930177] Lustre: server umount lustre-MDT0001 complete [ 8479.147393] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 8481.801082] Key type lgssc unregistered [ 8482.046980] LNet: 266578:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8482.055796] LNetError: 266578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8482.066945] LNet: Removed LNI 192.168.204.140@tcp [ 8482.822868] Key type .llcrypt unregistered [ 8482.825369] Key type ._llcrypt unregistered [ 8501.618663] Key type ._llcrypt registered [ 8501.622373] Key type .llcrypt registered [ 8501.772781] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 8502.811916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8502.847939] alg: No test for adler32 (adler32-zlib) [ 8504.024691] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8504.332063] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 8506.031440] Key type lgssc registered [ 8507.159633] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8515.727602] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8515.758881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8517.194181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8522.084127] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 8527.332449] LustreError: 267799:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8527.346403] LustreError: 267799:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8532.228119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8532.448929] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8532.631438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8537.861342] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 8546.841034] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8553.301748] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8559.446112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8559.767418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8561.848527] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8566.411361] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 8574.142625] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8592.866045] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8592.883914] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8592.919036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8593.900556] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8593.900600] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8593.924617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8593.924626] Lustre: Skipped 1 previous similar message [ 8598.007564] Lustre: server umount lustre-OST0000 complete [ 8604.129111] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8604.132706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8604.139038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8604.159481] Lustre: Skipped 1 previous similar message [ 8607.599922] Lustre: server umount lustre-MDT0000 complete [ 8609.256280] LustreError: 268981:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8609.276876] LustreError: 268981:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8611.395313] LustreError: 267778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449587 with bad export cookie 4545037088220045434 [ 8611.398146] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8611.409908] LustreError: 267778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8611.945219] Lustre: server umount lustre-MDT0001 complete [ 8623.263394] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 8625.833031] Key type lgssc unregistered [ 8626.086840] LNet: 271485:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8626.095292] LNetError: 271485:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8627.115208] LNet: Removed LNI 192.168.204.140@tcp [ 8627.971497] Key type .llcrypt unregistered [ 8627.977456] Key type ._llcrypt unregistered [ 8652.861361] Key type ._llcrypt registered [ 8652.863746] Key type .llcrypt registered [ 8652.969512] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 8654.535181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8654.570339] alg: No test for adler32 (adler32-zlib) [ 8655.605668] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8655.941873] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 8657.655597] Key type lgssc registered [ 8658.663249] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8666.627342] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8666.671122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8668.380969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8672.720405] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 8677.869188] LustreError: 272904:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8677.910024] LustreError: 272904:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8682.994648] LustreError: 272905:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8685.166907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8685.792247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8692.064943] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 8706.512474] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8714.038739] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8721.540630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8722.043684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8727.962188] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 8735.605813] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8737.393846] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8767.457104] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8767.466964] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8767.492696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8768.484859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8768.488860] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8768.488958] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8769.769703] Lustre: server umount lustre-OST0000 complete [ 8773.604445] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8773.627429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8773.635026] Lustre: Skipped 1 previous similar message [ 8778.209148] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8778.217651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8778.232491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8779.703170] Lustre: server umount lustre-MDT0000 complete [ 8783.707825] LustreError: 273639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449759 with bad export cookie 1556081589376376596 [ 8783.727078] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8784.117611] Lustre: server umount lustre-MDT0001 complete [ 8795.047652] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 8798.519892] Key type lgssc unregistered [ 8798.819628] LNet: 277214:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8798.826430] LNetError: 277214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8798.840217] LNet: Removed LNI 192.168.204.140@tcp [ 8799.790575] Key type .llcrypt unregistered [ 8799.794762] Key type ._llcrypt unregistered [ 8819.792482] Key type ._llcrypt registered [ 8819.796392] Key type .llcrypt registered [ 8820.016438] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 8821.676146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8821.817065] alg: No test for adler32 (adler32-zlib) [ 8823.234991] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8823.548331] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 8825.279208] Key type lgssc registered [ 8827.066677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8836.431528] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8836.500259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8838.234858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8843.403271] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 8848.363370] LustreError: 278449:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8848.384692] LustreError: 278449:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8852.517308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8852.877950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8857.378981] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 8864.696679] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8870.267688] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8876.122784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8876.426423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8882.065982] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 8882.627789] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8889.310274] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8903.141807] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8903.146398] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8903.162350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8903.164658] LustreError: Skipped 1 previous similar message [ 8903.176761] Lustre: Skipped 1 previous similar message [ 8906.754512] Lustre: server umount lustre-OST0000 complete [ 8913.378941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8913.391083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8913.411234] Lustre: Skipped 2 previous similar messages [ 8913.430299] Lustre: Skipped 1 previous similar message [ 8917.207640] Lustre: server umount lustre-MDT0000 complete [ 8918.504671] LustreError: 278452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8918.521797] LustreError: 278452:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8921.295831] LustreError: 278417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449896 with bad export cookie 8681468505810947465 [ 8921.302304] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8921.311409] LustreError: 278417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8921.713460] Lustre: server umount lustre-MDT0001 complete [ 8932.986302] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 8936.214639] Key type lgssc unregistered [ 8936.515730] LNet: 282153:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8936.538350] LNetError: 282153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8936.555720] LNet: Removed LNI 192.168.204.140@tcp [ 8937.449053] Key type .llcrypt unregistered [ 8937.452978] Key type ._llcrypt unregistered [ 8948.039221] Key type ._llcrypt registered [ 8948.042593] Key type .llcrypt registered [ 8948.211779] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8965.759038] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 8980.440899] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 8981.711324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8981.814217] alg: No test for adler32 (adler32-zlib) [ 8982.931785] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8983.195552] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 8984.905327] Key type lgssc registered [ 8986.170913] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9018.880703] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9018.947794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9020.197110] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9020.219730] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9020.293154] Lustre: lustre-MDT0000: new disk, initializing [ 9020.379585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9020.402609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9024.790847] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 9034.627622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9034.753978] Lustre: 285711:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9034.805334] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9034.808309] Lustre: Skipped 1 previous similar message [ 9034.875202] Lustre: lustre-MDT0001: new disk, initializing [ 9035.018761] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9035.072179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9035.089963] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9039.940718] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 9050.690727] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9057.312411] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9064.385541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9064.635592] Lustre: lustre-OST0000: new disk, initializing [ 9064.639982] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9064.646581] Lustre: 287678:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9064.779641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9072.054822] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 9072.769630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9072.778792] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9072.875891] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9082.699696] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9087.995618] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9088.252628] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9093.529746] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9093.831935] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9098.721777] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9098.723847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9098.738335] LustreError: Skipped 1 previous similar message [ 9098.739241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9098.777095] Lustre: Skipped 1 previous similar message [ 9102.259581] Lustre: server umount lustre-OST0000 complete [ 9106.917960] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9106.925417] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9106.942071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9106.951403] Lustre: Skipped 1 previous similar message [ 9108.961172] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9108.963458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9108.997529] Lustre: Skipped 1 previous similar message [ 9112.556449] Lustre: server umount lustre-MDT0000 complete [ 9116.124545] LustreError: 285702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450091 with bad export cookie 4003017643872050076 [ 9116.134340] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9116.145361] LustreError: 285702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9116.431811] Lustre: server umount lustre-MDT0001 complete [ 9125.642886] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 11:41:39 (1788450099) [ 9133.324591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9133.931893] LustreError: 290088:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9133.956064] LustreError: 290088:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9134.021947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9139.175547] LustreError: 290089:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9139.185746] LustreError: 290089:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9139.469516] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 9144.292439] LustreError: 290088:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9149.056961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9149.411906] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9149.541172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9154.692618] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 9164.045580] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9170.305640] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9179.148735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9179.580588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9187.039759] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 9196.049894] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9219.554429] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9219.565063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9219.573796] Lustre: Skipped 1 previous similar message [ 9219.596165] Lustre: Skipped 1 previous similar message [ 9223.293839] Lustre: server umount lustre-OST0000 complete [ 9229.796316] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9229.810171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9229.819210] Lustre: Skipped 1 previous similar message [ 9229.828732] Lustre: Skipped 1 previous similar message [ 9233.999546] Lustre: server umount lustre-MDT0000 complete [ 9234.913489] LustreError: 290083:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9234.951220] LustreError: 290083:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9238.269977] LustreError: 290071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450213 with bad export cookie 4003017643872050958 [ 9238.274851] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9238.282833] LustreError: 290071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9238.780591] Lustre: server umount lustre-MDT0001 complete [ 9251.754892] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 9255.235826] Key type lgssc unregistered [ 9255.672456] LNet: 293649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9255.695766] LNetError: 293649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9255.724345] LNet: Removed LNI 192.168.204.140@tcp [ 9256.775913] Key type .llcrypt unregistered [ 9256.784132] Key type ._llcrypt unregistered [ 9285.942445] Key type ._llcrypt registered [ 9285.945356] Key type .llcrypt registered [ 9286.059842] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 9302.927798] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 9304.561540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9304.594175] alg: No test for adler32 (adler32-zlib) [ 9305.741680] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 9306.073369] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 9307.855201] Key type lgssc registered [ 9309.414580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9361.813736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9361.888611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9363.197860] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9363.234512] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9363.387108] Lustre: lustre-MDT0000: new disk, initializing [ 9363.491969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9363.517604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9368.201371] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 9378.452107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9378.549279] Lustre: 297476:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9378.602874] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9378.606899] Lustre: Skipped 1 previous similar message [ 9378.707382] Lustre: lustre-MDT0001: new disk, initializing [ 9378.881249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9378.940459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9378.953595] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9384.368851] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 9395.728569] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9403.442979] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9410.927584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9411.205936] Lustre: lustre-OST2710: new disk, initializing [ 9411.209527] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9411.215703] Lustre: 299446:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9411.308180] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9418.830331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9418.840275] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9418.998717] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9419.449706] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 9430.319306] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9437.554146] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9437.717383] Lustre: lustre-OST03e8: new disk, initializing [ 9437.721805] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9437.730091] Lustre: 300511:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9437.826658] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9443.965606] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 9446.051896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9446.065412] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9446.163164] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9450.971658] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9457.045762] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9457.272646] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9461.519880] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9461.829521] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9467.191593] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9467.441545] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9472.471607] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9472.701687] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9477.092723] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9477.106704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9477.124683] Lustre: Skipped 1 previous similar message [ 9477.146878] Lustre: Skipped 3 previous similar messages [ 9482.219896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9482.234216] Lustre: Skipped 3 previous similar messages [ 9483.023708] Lustre: server umount lustre-MDT0000 complete [ 9487.052852] LustreError: 297470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450462 with bad export cookie 16269317897920575029 [ 9487.061154] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9487.335307] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9487.343218] Lustre: Skipped 2 previous similar messages [ 9487.529203] Lustre: server umount lustre-MDT0001 complete [ 9493.876976] Lustre: server umount lustre-OST2710 complete [ 9497.598049] Lustre: server umount lustre-OST03e8 complete [ 9519.451240] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 9531.343739] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 9569.673549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9569.956272] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9570.001733] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9570.069856] Lustre: lustre-MDT0000: new disk, initializing [ 9570.139543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9570.163706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9574.915292] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 9586.735741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9586.885362] Lustre: 305758:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9586.942990] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9586.946585] Lustre: Skipped 1 previous similar message [ 9587.013414] Lustre: lustre-MDT0001: new disk, initializing [ 9587.100795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9587.144924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9587.164846] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9591.548668] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 9602.078509] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9608.366607] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9615.813285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9616.167851] Lustre: lustre-OST0000: new disk, initializing [ 9616.171593] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9616.177826] Lustre: 307728:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9616.269466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9622.473799] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 9625.686120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9625.702595] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9625.872339] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9633.231642] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9639.079562] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9639.338871] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9644.371400] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9644.651559] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9651.682388] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9651.702982] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9651.707701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9651.711555] Lustre: Skipped 2 previous similar messages [ 9651.722316] Lustre: Skipped 1 previous similar message [ 9652.901950] Lustre: server umount lustre-OST0000 complete [ 9658.848729] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9658.861341] LustreError: Skipped 1 previous similar message [ 9658.870860] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9658.883543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9663.374913] Lustre: server umount lustre-MDT0000 complete [ 9667.044780] LustreError: 305771:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9667.069028] LustreError: 305771:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 9667.614636] LustreError: 305751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450643 with bad export cookie 16269317897920576709 [ 9667.617163] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9667.623699] LustreError: 305751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9667.885920] Lustre: server umount lustre-MDT0001 complete [ 9677.342754] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:50:51 (1788450651) [ 9678.975899] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9680.873911] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:50:55 (1788450655) [ 9685.211508] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 9715.260686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9715.453487] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9715.505044] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9715.622556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9720.809406] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 9732.503750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9732.595861] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9732.635899] Lustre: 311588:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9736.922268] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 9761.087893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9761.443228] Lustre: Denying initial registration attempt from nid 192.168.204.140@tcp, specified as failover [ 9761.455744] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9761.465778] LustreError: 313042:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9761.480495] LustreError: 313042:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9761.487283] LustreError: 313042:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9761.611295] Lustre: server umount lustre-OST0000 complete [ 9761.618710] LustreError: 313042:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9774.055390] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9774.069405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9774.088243] Lustre: Skipped 2 previous similar messages [ 9774.088636] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9774.116626] Lustre: Skipped 3 previous similar messages [ 9776.039275] Lustre: server umount lustre-MDT0000 complete [ 9779.177413] LustreError: 313045:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9779.204121] LustreError: 313045:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9780.409945] LustreError: 311581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450756 with bad export cookie 16269317897920577584 [ 9780.412048] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9780.430852] LustreError: 311581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9780.795255] Lustre: server umount lustre-MDT0001 complete [ 9792.226810] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 9795.056938] Key type lgssc unregistered [ 9795.338608] LNet: 314166:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9795.349032] LNetError: 314166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9795.366814] LNet: Removed LNI 192.168.204.140@tcp [ 9796.340451] Key type .llcrypt unregistered [ 9796.343583] Key type ._llcrypt unregistered [ 9809.696414] Key type ._llcrypt registered [ 9809.700501] Key type .llcrypt registered [ 9809.861336] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 9811.025596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9811.073109] alg: No test for adler32 (adler32-zlib) [ 9812.141616] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 9812.341099] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 9814.111730] Key type lgssc registered [ 9815.343385] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9860.159787] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 9869.896818] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9869.922741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9871.287654] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9871.301639] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9871.446773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9876.502854] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 9886.146389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9886.248594] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9886.315168] Lustre: 316931:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9886.644979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9892.005060] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 9900.028082] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9906.122864] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9925.887789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9926.180872] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9926.428318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9932.941763] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 9941.253505] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9946.593170] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9946.611096] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9946.624262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9947.108097] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9947.131831] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9947.149473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9947.155158] Lustre: Skipped 1 previous similar message [ 9949.802428] Lustre: server umount lustre-OST0000 complete [ 9957.349555] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9957.372265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9957.389463] Lustre: Skipped 1 previous similar message [ 9959.909718] Lustre: server umount lustre-MDT0000 complete [ 9962.467046] LustreError: 316942:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9962.498129] LustreError: 316942:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9963.582992] LustreError: 317671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450939 with bad export cookie 4285658803990847960 [ 9963.584314] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9963.590899] LustreError: 317671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9963.920502] Lustre: server umount lustre-MDT0001 complete [ 9975.173960] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 9977.717568] Key type lgssc unregistered [ 9978.073909] LNet: 320508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9978.086746] LNetError: 320508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9978.103246] LNet: Removed LNI 192.168.204.140@tcp [ 9978.957487] Key type .llcrypt unregistered [ 9978.961409] Key type ._llcrypt unregistered [10005.501086] Key type ._llcrypt registered [10005.505919] Key type .llcrypt registered [10005.660483] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [10007.394034] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10007.426487] alg: No test for adler32 (adler32-zlib) [10008.607746] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [10008.978277] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [10010.703279] Key type lgssc registered [10011.987752] Lustre: Echo OBD driver; http://www.lustre.org/ [10020.876844] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10020.909524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10022.564950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10027.742862] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [10033.122103] LustreError: 321925:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10033.142104] LustreError: 321925:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10038.008931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10038.240518] LustreError: 321926:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10038.494283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10043.563381] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [10053.293096] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10059.930443] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10070.251685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10070.748470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10078.530920] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [10086.835657] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10109.053810] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.40@tcp (stopping) [10109.927460] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10109.932733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10109.950074] Lustre: Skipped 2 previous similar messages [10109.969470] Lustre: Skipped 2 previous similar messages [10114.146802] LustreError: 321920:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10114.243587] Lustre: server umount lustre-MDT0000 complete [10117.209996] LustreError: 324224:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10117.240360] LustreError: 324224:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10118.968922] LustreError: 321905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451094 with bad export cookie 6923567363438305026 [10118.971477] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10118.987465] LustreError: 321905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10120.165938] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10120.195060] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10122.325229] LustreError: 321921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10122.355484] LustreError: 321921:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10122.366275] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.40@tcp (stopping) [10125.408778] Lustre: server umount lustre-MDT0001 complete [10132.706229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10142.178831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10144.671954] LustreError: 321235:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000003d6785b2 x1875326779040896/t0(0) o250->MGC192.168.204.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10145.080824] LustreError: 323757:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10145.093499] LustreError: 323757:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10145.161270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10145.221206] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10150.410346] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [10153.394180] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10159.389073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10159.809324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10164.395902] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [10165.235340] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10172.529240] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10174.236473] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10180.488766] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10182.094392] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10190.818993] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10190.823705] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10190.830512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10190.830520] Lustre: Skipped 1 previous similar message [10192.926737] Lustre: server umount lustre-OST0000 complete [10200.549250] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10200.556673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10200.577719] Lustre: Skipped 1 previous similar message [10200.585517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10200.589401] Lustre: Skipped 1 previous similar message [10211.295495] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10211.474341] Lustre: server umount lustre-MDT0000 complete [10215.399569] LustreError: 321905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451191 with bad export cookie 6923567363438312516 [10215.405444] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10215.415626] LustreError: 321905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10215.882225] Lustre: server umount lustre-MDT0001 complete [10227.593863] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [10229.784917] Key type lgssc unregistered [10230.046161] LNet: 328218:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10230.060316] LNetError: 328218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10230.077281] LNet: Removed LNI 192.168.204.140@tcp [10230.981676] Key type .llcrypt unregistered [10230.983537] Key type ._llcrypt unregistered [10261.048700] Key type ._llcrypt registered [10261.054994] Key type .llcrypt registered [10261.163669] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [10262.035476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10262.062730] alg: No test for adler32 (adler32-zlib) [10263.227821] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [10263.534260] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [10265.208096] Key type lgssc registered [10266.619132] Lustre: Echo OBD driver; http://www.lustre.org/ [10275.431514] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10275.454850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10276.843795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10281.067887] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [10287.078551] LustreError: 329638:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10287.094514] LustreError: 329638:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10290.976387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10291.428745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10296.841889] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [10305.664616] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10312.506561] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10317.165396] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10317.174042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10317.179470] Lustre: Skipped 1 previous similar message [10317.188446] Lustre: Skipped 1 previous similar message [10322.405764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10322.423521] Lustre: Skipped 1 previous similar message [10322.714869] Lustre: server umount lustre-MDT0000 complete [10326.936022] LustreError: 329619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451302 with bad export cookie 618715402419753928 [10326.938217] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10326.947290] LustreError: 329619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10327.356645] Lustre: server umount lustre-MDT0001 complete [10343.988560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10344.307631] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10344.350956] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10344.631529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10349.903781] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [10358.390640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10358.437668] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10358.456301] Lustre: 332239:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10358.879542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10364.285722] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [10373.931269] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10380.607251] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10391.344319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10391.639683] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10391.654210] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10391.672698] LustreError: 334122:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10391.688786] LustreError: 334122:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10391.700838] LustreError: 334122:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10391.808759] Lustre: server umount lustre-OST0000 complete [10391.819693] LustreError: 334122:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [10403.698648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10403.987986] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10404.338507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10409.635644] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10411.699154] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [10420.074671] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10428.828604] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10428.976067] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10429.087735] Lustre: lustre-OST0001: new disk, initializing [10429.094681] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10429.102278] Lustre: 335662:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10429.247729] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10436.830343] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [10438.958859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10438.998855] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10439.172320] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10445.721883] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10454.497707] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10454.502280] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10454.521941] LustreError: Skipped 1 previous similar message [10454.528380] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10454.554073] Lustre: Skipped 1 previous similar message [10454.596289] Lustre: Skipped 1 previous similar message [10455.980356] Lustre: server umount lustre-OST0001 complete [10459.628431] LustreError: 334588:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10459.663945] LustreError: 334588:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10461.154693] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10461.162111] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10461.179764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10464.742570] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10464.744052] LustreError: 334588:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10464.761307] LustreError: 334588:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10467.484820] Lustre: server umount lustre-OST0000 complete [10474.980997] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10474.986321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10475.008115] Lustre: Skipped 1 previous similar message [10475.020121] Lustre: Skipped 3 previous similar messages [10478.363913] Lustre: server umount lustre-MDT0000 complete [10480.116735] LustreError: 336285:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10480.139694] LustreError: 336285:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10482.799424] LustreError: 332230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451458 with bad export cookie 618715402419754474 [10482.813512] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10482.819619] LustreError: 332230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10483.595522] Lustre: server umount lustre-MDT0001 complete [10495.987598] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [10499.212150] Key type lgssc unregistered [10499.580192] LNet: 337626:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10499.589377] LNetError: 337626:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10499.623252] LNet: Removed LNI 192.168.204.140@tcp [10500.680196] Key type .llcrypt unregistered [10500.688256] Key type ._llcrypt unregistered [10578.423410] Key type ._llcrypt registered [10578.426305] Key type .llcrypt registered [10578.527040] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [10601.954774] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [10603.956718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10603.969943] alg: No test for adler32 (adler32-zlib) [10605.125451] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [10605.666739] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [10607.615193] Key type lgssc registered [10610.355360] Lustre: Echo OBD driver; http://www.lustre.org/ [10649.608345] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10649.684247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10650.945037] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10650.983897] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10651.084935] Lustre: lustre-MDT0000: new disk, initializing [10651.270800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10651.302446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10656.668725] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [10668.565048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10668.776539] Lustre: 342567:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10668.889582] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10668.895183] Lustre: Skipped 1 previous similar message [10668.900453] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10668.980624] Lustre: lustre-MDT0001: new disk, initializing [10669.115020] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10669.171693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10669.193643] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10674.277377] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [10685.119816] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10692.731553] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10700.933732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10701.461964] Lustre: lustre-OST0000: new disk, initializing [10701.473596] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10701.482109] Lustre: 344533:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10701.603052] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10709.158429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10709.184714] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10709.345390] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10709.803884] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [10721.525180] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10727.876394] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10728.244698] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10734.530759] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10734.787971] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10740.197074] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10740.199908] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10740.203578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10740.223600] LustreError: Skipped 1 previous similar message [10740.274705] Lustre: Skipped 1 previous similar message [10743.518651] Lustre: server umount lustre-OST0000 complete [10750.434950] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10750.438868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10750.444748] Lustre: Skipped 1 previous similar message [10750.461023] Lustre: Skipped 2 previous similar messages [10754.362512] Lustre: server umount lustre-MDT0000 complete [10759.027269] LustreError: 342557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451734 with bad export cookie 1512950209084987266 [10759.033540] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10759.049491] LustreError: 342557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10759.709948] Lustre: server umount lustre-MDT0001 complete [10769.878119] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 12:09:03 (1788451743) [10781.287715] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 12:09:14 (1788451754) [10791.360634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10791.730262] LustreError: 347293:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10791.746332] LustreError: 347293:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10791.786591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10797.032882] LustreError: 347294:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10797.171179] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [10802.149187] LustreError: 347293:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10807.259517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10807.272299] LustreError: 347294:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10807.707764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10812.615225] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [10821.474438] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10828.809344] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10836.439963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10836.809344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10843.618244] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [10852.523219] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10871.468931] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10873.706902] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10875.760671] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10877.874121] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10880.135960] Lustre: Failing over lustre-MDT0000 [10880.486082] Lustre: server umount lustre-MDT0000 complete [10883.041644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10883.056778] LustreError: 347290:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10883.058453] Lustre: Skipped 2 previous similar messages [10883.083678] LustreError: 347290:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10893.285490] LustreError: 347294:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10893.302216] LustreError: 347294:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [10898.399544] Lustre: 340788:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788451858/real 1788451858] req@00000000c05fb7ce x1875327405077376/t0(0) o400->MGC192.168.204.140@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788451874 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10898.440256] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10900.548328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10908.652318] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x14ff1432ff5d1f0e [10908.674849] Lustre: MGC192.168.204.140@tcp: Connection restored to 0@lo (at 0@lo) [10908.992644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10909.052472] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10911.835535] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10913.862379] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [10914.293683] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10914.328655] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10914.393865] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10924.585614] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10926.292208] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10929.989334] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10931.918447] Lustre: Failing over lustre-MDT0000 [10932.225836] Lustre: server umount lustre-MDT0000 complete [10932.350928] LustreError: 348450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10932.380841] LustreError: 348450:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [10934.752270] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10934.757217] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10934.788607] Lustre: Skipped 2 previous similar messages [10951.136436] Lustre: 340787:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788451910/real 1788451910] req@000000002d6618c3 x1875327405098240/t0(0) o400->MGC192.168.204.140@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788451926 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10951.184165] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10952.722076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10961.376682] LustreError: 340786:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d84ffdac x1875327405103744/t0(0) o250->MGC192.168.204.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10961.779472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10961.925701] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10964.048674] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10967.041611] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10967.051553] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10967.056202] Lustre: Skipped 2 previous similar messages [10967.164700] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10968.969250] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [10979.312275] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10981.319082] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10992.619937] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10992.620768] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10992.626884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10992.647155] Lustre: Skipped 2 previous similar messages [10994.382135] Lustre: server umount lustre-MDT0000 complete [10997.730652] LustreError: 348289:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10997.753359] LustreError: 348289:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 29 previous similar messages [10999.242310] LustreError: 347275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451974 with bad export cookie 1512950209084989982 [10999.248553] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10999.254666] LustreError: 347275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10999.962651] Lustre: server umount lustre-MDT0001 complete [11016.101148] Lustre: server umount lustre-OST0000 complete [11037.425981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11038.039655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11043.416263] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [11051.400735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11056.027985] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [11063.245972] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11069.787106] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11077.692325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11083.221412] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11085.950181] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [11094.586423] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11106.796692] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11106.802652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11106.802812] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11106.821988] Lustre: Skipped 1 previous similar message [11106.863935] Lustre: Skipped 3 previous similar messages [11107.796749] Lustre: server umount lustre-OST0000 complete [11113.440353] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11117.025912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11117.031426] Lustre: Skipped 2 previous similar messages [11118.860569] Lustre: server umount lustre-MDT0000 complete [11122.146178] LustreError: 354390:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11122.179280] LustreError: 354390:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11123.331807] LustreError: 353642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452098 with bad export cookie 1512950209084990724 [11123.335166] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11123.348395] LustreError: 353642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11123.682438] Lustre: server umount lustre-MDT0001 complete [11134.615477] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [11137.344989] Key type lgssc unregistered [11137.685655] LNet: 357200:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11137.692556] LNetError: 357200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11137.715901] LNet: Removed LNI 192.168.204.140@tcp [11138.533723] Key type .llcrypt unregistered [11138.536320] Key type ._llcrypt unregistered [11166.041539] Key type ._llcrypt registered [11166.045685] Key type .llcrypt registered [11166.245358] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [11167.484703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11167.556892] alg: No test for adler32 (adler32-zlib) [11168.638682] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11168.891580] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [11170.503206] Key type lgssc registered [11171.633954] Lustre: Echo OBD driver; http://www.lustre.org/ [11182.101329] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11182.129325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11184.010452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11189.567800] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [11194.372796] LustreError: 358718:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11194.429830] LustreError: 358718:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11199.459044] LustreError: 358719:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11200.244420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11200.661534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11205.498956] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [11215.095526] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11222.992915] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11231.096625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11231.551667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11239.122775] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [11248.275573] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11254.083617] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11267.040499] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11267.050206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11267.067649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11269.600527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11269.601932] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11269.616173] Lustre: Skipped 1 previous similar message [11271.125063] Lustre: server umount lustre-MDT0000 complete [11274.720697] LustreError: 358719:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11274.742643] LustreError: 358719:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11274.770305] LustreError: 359454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452250 with bad export cookie 6647307955407471919 [11274.771109] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11274.796748] LustreError: 359454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11275.118838] Lustre: server umount lustre-MDT0001 complete [11290.165880] Lustre: server umount lustre-OST0000 complete [11304.627209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11305.230322] LustreError: 362455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11305.302220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11310.285786] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [11310.564945] LustreError: 362456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11318.838393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11319.267574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11323.387838] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [11330.535457] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11338.833164] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11347.156232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11347.556901] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11349.640621] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11355.041347] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [11363.479352] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11379.687865] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11379.696311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11379.712613] Lustre: Skipped 2 previous similar messages [11379.731636] Lustre: Skipped 4 previous similar messages [11382.596533] Lustre: server umount lustre-MDT0000 complete [11384.816574] LustreError: 362452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11384.852616] LustreError: 362452:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11387.110519] LustreError: 362436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452362 with bad export cookie 6647307955407473032 [11387.125087] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11387.882939] Lustre: server umount lustre-MDT0001 complete [11403.160788] Lustre: server umount lustre-OST0000 complete [11419.897669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11420.506694] LustreError: 366366:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11420.588796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11424.930210] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [11433.924349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11439.491077] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [11447.341584] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11453.189893] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11460.411721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11460.690329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11460.693831] Lustre: Skipped 1 previous similar message [11465.849373] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11466.900475] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [11474.259426] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11481.568676] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11481.574289] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11481.612127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11484.641958] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11485.686899] Lustre: server umount lustre-OST0000 complete [11490.785362] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11490.798451] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11490.823614] Lustre: Skipped 1 previous similar message [11490.833256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11490.838129] Lustre: Skipped 2 previous similar messages [11496.317898] Lustre: server umount lustre-MDT0000 complete [11500.000377] LustreError: 366366:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11500.029425] LustreError: 366366:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11501.384395] LustreError: 366348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452477 with bad export cookie 6647307955407474054 [11501.405430] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11501.412085] LustreError: 366348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [11502.158533] Lustre: server umount lustre-MDT0001 complete [11514.759282] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [11518.937315] Key type lgssc unregistered [11519.327163] LNet: 369903:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11519.350218] LNetError: 369903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11519.386660] LNet: Removed LNI 192.168.204.140@tcp [11520.455512] Key type .llcrypt unregistered [11520.458933] Key type ._llcrypt unregistered [11547.622700] Key type ._llcrypt registered [11547.624691] Key type .llcrypt registered [11547.713710] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [11548.688975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11548.843038] alg: No test for adler32 (adler32-zlib) [11549.926608] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11550.246875] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [11551.929050] Key type lgssc registered [11552.957659] Lustre: Echo OBD driver; http://www.lustre.org/ [11560.999069] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11561.035890] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11561.046853] LustreError: 371342:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11561.064898] LustreError: 371342:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11561.080751] LustreError: 371342:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11561.088867] LustreError: 371342:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11561.102622] LustreError: 371342:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [11569.235678] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11569.241509] LustreError: 371740:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11569.267655] LustreError: 371740:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11569.277150] LustreError: 371740:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11569.287699] LustreError: 371740:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11569.297843] LustreError: 371740:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [11590.438462] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [11594.223930] Key type lgssc unregistered [11594.658653] LNet: 372748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11594.683348] LNetError: 372748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11594.731151] LNet: Removed LNI 192.168.204.140@tcp [11596.074409] Key type .llcrypt unregistered [11596.076841] Key type ._llcrypt unregistered [11619.812706] Key type ._llcrypt registered [11619.822259] Key type .llcrypt registered [11620.047514] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [11639.328718] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [11640.806721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11640.845729] alg: No test for adler32 (adler32-zlib) [11641.901457] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11642.192313] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [11643.911311] Key type lgssc registered [11645.307468] Lustre: Echo OBD driver; http://www.lustre.org/ [11691.502338] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11691.578227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11692.808914] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11692.864395] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11692.989382] Lustre: lustre-MDT0000: new disk, initializing [11693.200158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11693.243818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11699.860400] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [11714.546827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11714.765474] Lustre: 375910:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11714.841822] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11714.848758] Lustre: Skipped 1 previous similar message [11714.947215] Lustre: lustre-MDT0001: new disk, initializing [11715.080427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11715.141068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11715.152620] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11721.111865] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [11733.860561] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11741.188273] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11749.687515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11750.059623] Lustre: lustre-OST0000: new disk, initializing [11750.080294] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11750.098074] Lustre: 377879:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11750.244691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11757.903032] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [11758.697323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11758.705234] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11758.961120] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11768.703322] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11774.064214] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11774.455510] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11779.988416] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11780.234729] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11784.683622] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11784.689962] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11784.693481] LustreError: Skipped 1 previous similar message [11784.717956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11784.722122] Lustre: Skipped 1 previous similar message [11788.983881] Lustre: server umount lustre-OST0000 complete [11794.912509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11794.940757] Lustre: Skipped 1 previous similar message [11794.947613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11794.957948] Lustre: Skipped 1 previous similar message [11800.034512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11800.044201] Lustre: Skipped 1 previous similar message [11800.432969] Lustre: server umount lustre-MDT0000 complete [11804.687305] LustreError: 377878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452780 with bad export cookie 18241377399308690866 [11804.691765] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11804.698649] LustreError: 377878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11805.317855] Lustre: server umount lustre-MDT0001 complete [11816.497903] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11818.271650] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 12:26:32 (1788452792) [11826.417686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11827.310311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11832.014738] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [11832.293206] LustreError: 380340:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11832.310229] LustreError: 380340:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11836.401318] LustreError: 380339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11841.505948] LustreError: 380340:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11842.966407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11843.809379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11850.863774] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [11860.806812] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11868.020685] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11875.672924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11876.183462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11882.642293] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [11890.942874] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11899.613055] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11899.900381] Lustre: lustre-OST0001: new disk, initializing [11899.905353] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11899.919232] Lustre: 383177:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11900.120339] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11907.309900] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [11908.294246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11908.333140] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11908.534973] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11919.378300] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11923.938906] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11923.943944] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11923.952654] Lustre: Skipped 1 previous similar message [11923.954418] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11923.957686] LustreError: Skipped 1 previous similar message [11923.959673] Lustre: Skipped 1 previous similar message [11929.073729] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11929.089817] Lustre: Skipped 2 previous similar messages [11929.971840] Lustre: server umount lustre-OST0001 complete [11931.749313] LustreError: 383566:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11939.305254] LustreError: 382164:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11939.320546] LustreError: 382164:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11949.542780] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11949.547776] LustreError: 382166:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11949.552469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11949.555967] Lustre: Skipped 2 previous similar messages [11949.575117] LustreError: 382166:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11962.339675] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11962.549131] Lustre: server umount lustre-OST0000 complete [11970.017149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11970.018665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11970.038607] Lustre: Skipped 1 previous similar message [11970.047327] Lustre: Skipped 5 previous similar messages [11973.053306] Lustre: server umount lustre-MDT0000 complete [11975.140740] LustreError: 381438:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11975.157945] LustreError: 381438:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11977.049681] LustreError: 381074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452952 with bad export cookie 18241377399308691748 [11977.051430] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11977.059020] LustreError: 381074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11983.500947] Lustre: server umount lustre-MDT0001 complete [11995.061739] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [11997.840825] Key type lgssc unregistered [11998.206167] LNet: 385173:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11998.217150] LNetError: 385173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11998.235055] LNet: Removed LNI 192.168.204.140@tcp [11999.284452] Key type .llcrypt unregistered [11999.288266] Key type ._llcrypt unregistered [12050.317308] Key type ._llcrypt registered [12050.320579] Key type .llcrypt registered [12050.437563] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [12051.555653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12051.642301] alg: No test for adler32 (adler32-zlib) [12052.747315] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12052.970673] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [12054.671181] Key type lgssc registered [12055.599804] Lustre: Echo OBD driver; http://www.lustre.org/ [12063.276170] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12063.288924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12064.550673] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12064.577028] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12064.784187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12068.626861] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [12077.742811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12077.916753] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12077.986093] Lustre: 387975:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12078.570030] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12085.288967] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [12099.249227] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12106.158519] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12109.288502] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12109.295154] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12109.307811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12113.889803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12113.890614] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12115.908974] Lustre: server umount lustre-MDT0000 complete [12120.798444] LustreError: 387966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453096 with bad export cookie 16709196757051346995 [12120.808632] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12120.811498] LustreError: 387966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12121.036619] Lustre: server umount lustre-MDT0001 complete [12132.906432] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 12:31:47 (1788453107) [12140.202234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12140.844710] LustreError: 390488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12140.870949] LustreError: 390488:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12140.921861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12146.150861] LustreError: 390489:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12146.370884] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [12151.265904] LustreError: 390488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12155.811367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12156.123326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12161.121444] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [12168.268606] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12174.547550] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12182.732829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12182.896784] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12183.111408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12190.453519] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [12200.819862] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12212.246793] Lustre: Permanently deactivating lustre-OST0000 [12212.282066] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12220.484192] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12223.494884] LustreError: 393478:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12223.507465] LustreError: 393478:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12223.514754] LustreError: 393478:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12226.465742] LustreError: 393526:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12226.476844] LustreError: 393526:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12235.228276] Lustre: server umount lustre-OST0000 complete [12238.638977] LustreError: 393776:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12238.650135] LustreError: 393776:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12242.400607] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12242.408210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12242.429180] Lustre: Skipped 1 previous similar message [12242.450906] Lustre: Skipped 2 previous similar messages [12247.521789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12247.538116] Lustre: Skipped 1 previous similar message [12250.820851] Lustre: server umount lustre-MDT0000 complete [12252.642346] LustreError: 390484:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12252.672955] LustreError: 390484:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12256.081202] LustreError: 390469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453231 with bad export cookie 16709196757051347569 [12256.085462] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12256.093955] LustreError: 390469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12265.610577] Lustre: server umount lustre-MDT0001 complete [12276.844550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12281.928293] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [12285.174609] LustreError: 394935:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12293.373729] Lustre: 395274:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.140@tcp [12296.110056] Lustre: server umount MGS complete [12308.548099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12308.950958] LustreError: 395999:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12308.967896] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12308.985921] Lustre: Skipped 1 previous similar message [12309.060532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12314.320158] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [12319.204366] LustreError: 395999:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12319.223821] LustreError: 395999:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12323.726062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12324.141559] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12328.966804] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [12337.683341] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12344.677540] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12347.308502] Lustre: Permanently reactivating lustre-OST0000 [12347.323647] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12347.337560] Lustre: Skipped 2 previous similar messages [12373.476423] LustreError: 397735:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12382.814675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12383.096951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12383.104963] Lustre: Skipped 1 previous similar message [12389.613676] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [12398.585526] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12408.802141] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12408.813332] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12408.826855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12413.848219] Lustre: server umount lustre-OST0000 complete [12420.075045] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12420.099115] Lustre: Skipped 1 previous similar message [12420.120342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12420.128414] Lustre: Skipped 3 previous similar messages [12433.377303] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12433.771873] Lustre: server umount lustre-MDT0000 complete [12434.401205] LustreError: 397006:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12434.421727] LustreError: 397006:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12438.267453] LustreError: 397577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453413 with bad export cookie 16709196757051348626 [12438.272104] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12438.278865] LustreError: 397577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12438.670614] Lustre: server umount lustre-MDT0001 complete [12451.508625] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [12454.695875] Key type lgssc unregistered [12455.131213] LNet: 400325:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12455.136582] LNetError: 400325:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12455.167905] LNet: Removed LNI 192.168.204.140@tcp [12456.089512] Key type .llcrypt unregistered [12456.091521] Key type ._llcrypt unregistered [12479.334824] Key type ._llcrypt registered [12479.339480] Key type .llcrypt registered [12479.550667] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [12498.031759] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [12499.460257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12499.609200] alg: No test for adler32 (adler32-zlib) [12500.846228] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12501.167497] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [12502.871207] Key type lgssc registered [12504.080111] Lustre: Echo OBD driver; http://www.lustre.org/ [12548.235527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12548.275614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12549.607840] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12549.632100] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12549.704517] Lustre: lustre-MDT0000: new disk, initializing [12549.785993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12549.803143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12555.709771] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [12569.007839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12569.244915] Lustre: 403488:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12569.353914] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12569.368947] Lustre: Skipped 1 previous similar message [12569.506758] Lustre: lustre-MDT0001: new disk, initializing [12569.640789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12569.732157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12569.752688] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12574.972691] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [12586.001617] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12592.854855] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12600.681619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12601.143560] Lustre: lustre-OST0000: new disk, initializing [12601.164175] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12601.172867] Lustre: 405458:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12601.298524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12607.826342] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [12608.598866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12608.616631] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12608.758406] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12619.636848] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12624.713452] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12625.195074] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12631.980626] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12632.756436] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12639.716077] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12639.717109] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12639.759332] LustreError: Skipped 1 previous similar message [12639.766651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12643.753295] Lustre: server umount lustre-OST0000 complete [12649.960899] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12649.988302] Lustre: Skipped 1 previous similar message [12649.995511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12650.005170] Lustre: Skipped 2 previous similar messages [12655.073939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12655.080451] Lustre: Skipped 1 previous similar message [12655.536364] Lustre: server umount lustre-MDT0000 complete [12660.338250] LustreError: 403478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453635 with bad export cookie 11164804790904599436 [12660.339197] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12660.351571] LustreError: 403478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12660.718552] Lustre: server umount lustre-MDT0001 complete [12671.547921] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12673.824342] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 12:40:47 (1788453647) [12681.736118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12682.275960] LustreError: 407923:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12682.298618] LustreError: 407923:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12682.341593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12687.843270] LustreError: 407924:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12688.364503] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [12691.948527] LustreError: 407924:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12697.065825] LustreError: 407923:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12698.600769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12699.002094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12704.430164] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [12712.724582] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12720.042896] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12728.350055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12728.826099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12736.826627] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [12745.772361] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12747.918166] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12754.773972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12762.605873] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12762.614577] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12762.638960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12762.672353] Lustre: Skipped 2 previous similar messages [12766.714420] Lustre: server umount lustre-OST0000 complete [12770.795149] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12770.804918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12770.818950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12770.823854] Lustre: Skipped 1 previous similar message [12783.077916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12783.082694] Lustre: Skipped 5 previous similar messages [12785.121687] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12785.284084] Lustre: server umount lustre-MDT0000 complete [12788.200614] LustreError: 407924:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12788.218651] LustreError: 407924:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12789.371835] LustreError: 407905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453765 with bad export cookie 11164804790904600318 [12789.383217] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12789.389683] LustreError: 407905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12789.831774] Lustre: server umount lustre-MDT0001 complete [12802.740170] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [12805.906562] Key type lgssc unregistered [12806.242951] LNet: 411654:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12806.259470] LNetError: 411654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12806.285682] LNet: Removed LNI 192.168.204.140@tcp [12807.272201] Key type .llcrypt unregistered [12807.276567] Key type ._llcrypt unregistered [12831.161361] Key type ._llcrypt registered [12831.165147] Key type .llcrypt registered [12831.257176] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [12831.534515] Key type .llcrypt unregistered [12831.538420] Key type ._llcrypt unregistered [12849.180953] Key type ._llcrypt registered [12849.184116] Key type .llcrypt registered [12849.327580] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [12850.461950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12850.514633] alg: No test for adler32 (adler32-zlib) [12851.699831] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12851.948210] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [12853.647333] Key type lgssc registered [12854.995159] Lustre: Echo OBD driver; http://www.lustre.org/ [12863.572355] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12863.588827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12865.062784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12870.290546] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [12875.243395] LustreError: 413873:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12875.259154] LustreError: 413873:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12880.042337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12880.361024] LustreError: 413872:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12880.546839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12885.985553] LustreError: 413873:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12886.004193] LustreError: 413873:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12886.675779] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [12890.084648] LustreError: 413869:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12890.103046] LustreError: 413869:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12890.119078] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12895.210748] LustreError: 413872:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12895.247832] LustreError: 413872:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12895.251988] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12902.188606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12902.605341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12907.807686] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [12918.248317] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12918.249970] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12918.261484] Lustre: Skipped 1 previous similar message [12918.262292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12919.767791] Lustre: server umount lustre-OST0000 complete [12924.316497] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12924.323605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12924.353502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12924.361886] Lustre: Skipped 1 previous similar message [12924.595826] Lustre: server umount lustre-MDT0000 complete [12928.481512] LustreError: 413873:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12928.501980] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12928.508723] LustreError: 413873:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12929.088076] LustreError: 413853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453904 with bad export cookie 12860087463438060769 [12929.093141] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12929.124572] LustreError: 413853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12929.478352] Lustre: server umount lustre-MDT0001 complete [12943.783891] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [12947.125283] Key type lgssc unregistered [12947.407953] LNet: 417022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12947.416735] LNetError: 417022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12947.431904] LNet: Removed LNI 192.168.204.140@tcp [12948.496189] Key type .llcrypt unregistered [12948.497957] Key type ._llcrypt unregistered [12974.604641] Key type ._llcrypt registered [12974.606766] Key type .llcrypt registered [12974.707476] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [12976.024839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12976.288858] alg: No test for adler32 (adler32-zlib) [12977.419964] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12977.802664] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [12979.615267] Key type lgssc registered [12981.240886] Lustre: Echo OBD driver; http://www.lustre.org/ [12990.047072] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12990.063893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13005.792509] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13010.912744] LustreError: 418414:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.140@tcp: failed processing log, type 4: rc = -110 [13037.669724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13045.867796] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [13055.361444] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13064.080988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13064.519020] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13064.603897] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13068.915386] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [13069.799711] LustreError: 418439:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13069.813639] LustreError: 418439:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13072.972296] LustreError: 419452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13073.013193] LustreError: 419452:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13077.989233] LustreError: 419454:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13078.039080] LustreError: 419454:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13080.129543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13080.834764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13086.516560] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [13097.585904] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13105.909563] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13113.740290] Lustre: server umount lustre-OST0000 complete [13119.977512] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13119.991697] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13120.019659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13122.019069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13122.024807] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13127.137422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13127.154288] Lustre: Skipped 2 previous similar messages [13128.242258] Lustre: server umount lustre-MDT0000 complete [13132.261524] LustreError: 420095:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13132.287479] LustreError: 420095:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13132.873058] LustreError: 418420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454108 with bad export cookie 17862657081404805640 [13132.896493] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13132.905231] LustreError: 418420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13142.633965] Lustre: server umount lustre-MDT0001 complete [13154.948269] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [13158.023422] Key type lgssc unregistered [13158.286074] LNet: 421992:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13158.291641] LNetError: 421992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13158.317639] LNet: Removed LNI 192.168.204.140@tcp [13159.147211] Key type .llcrypt unregistered [13159.148804] Key type ._llcrypt unregistered [13188.366353] Key type ._llcrypt registered [13188.370344] Key type .llcrypt registered [13188.507282] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [13189.855372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13189.922728] alg: No test for adler32 (adler32-zlib) [13191.165382] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [13191.499974] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [13193.175175] Key type lgssc registered [13195.012021] Lustre: Echo OBD driver; http://www.lustre.org/ [13204.333366] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13204.349507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13205.909841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13212.387820] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [13216.227380] LustreError: 423410:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13216.245556] LustreError: 423410:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13221.345630] LustreError: 423411:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13222.974384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13223.434628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13228.188655] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [13236.375105] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13242.576178] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13249.392298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13249.768372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13250.811367] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13256.110761] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [13265.548885] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13273.539434] Lustre: server umount lustre-MDT0000 complete [13273.574305] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13273.591765] LustreError: 424409:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13273.605671] Lustre: Skipped 2 previous similar messages [13273.656847] LustreError: 424409:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13276.446295] LustreError: 423393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.40@tcp arrived at 1788454252 with bad export cookie 4111336441268298497 [13276.457261] LustreError: 423393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13278.621440] LustreError: 423391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454254 with bad export cookie 4111336441268297608 [13278.632115] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13278.688379] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13278.691701] LustreError: 423405:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13278.706863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13278.723448] LustreError: 423405:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13283.813768] LustreError: 424410:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13283.815412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13283.845732] LustreError: 424410:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13284.895600] Lustre: server umount lustre-OST0000 complete [13291.589720] Lustre: server umount lustre-MDT0001 complete [13304.773736] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [13308.384444] Key type lgssc unregistered [13308.712325] LNet: 427098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13308.718618] LNetError: 427098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13308.732452] LNet: Removed LNI 192.168.204.140@tcp [13309.601823] Key type .llcrypt unregistered [13309.603782] Key type ._llcrypt unregistered [13338.628558] Key type ._llcrypt registered [13338.631328] Key type .llcrypt registered [13338.749644] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [13340.281575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13340.395359] alg: No test for adler32 (adler32-zlib) [13341.575760] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [13341.904545] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [13343.647163] Key type lgssc registered [13345.167670] Lustre: Echo OBD driver; http://www.lustre.org/ [13353.684257] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13353.707768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13355.365806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13360.623961] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [13364.707898] LustreError: 428514:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13364.722552] LustreError: 428514:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13369.827855] LustreError: 428515:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13369.912173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13370.342183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13375.029663] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [13383.664941] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13389.485473] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13396.564634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13397.008422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13400.117958] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13404.110741] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [13412.117634] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13416.421110] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13416.435463] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13416.451626] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13419.090981] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.40@tcp (stopping) [13419.501383] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13424.213032] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.40@tcp (stopping) [13424.231863] Lustre: Skipped 2 previous similar messages [13429.341893] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.40@tcp (stopping) [13429.353865] Lustre: Skipped 2 previous similar messages [13430.239214] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13430.500615] Lustre: server umount lustre-MDT0001 complete [13434.852722] LustreError: 428515:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13434.885558] LustreError: 428515:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13439.969243] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13439.975972] LustreError: 429242:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13439.979913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13439.979923] Lustre: Skipped 2 previous similar messages [13444.079939] Lustre: server umount lustre-OST0000 complete [13445.098548] LustreError: 428515:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13448.448624] Lustre: server umount lustre-MDT0000 complete [13464.109903] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [13467.430342] Key type lgssc unregistered [13467.713352] LNet: 432204:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13467.726511] LNetError: 432204:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13467.764704] LNet: Removed LNI 192.168.204.140@tcp [13468.711377] Key type .llcrypt unregistered [13468.713337] Key type ._llcrypt unregistered [13495.016292] Key type ._llcrypt registered [13495.019662] Key type .llcrypt registered [13495.118070] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [13512.188748] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [13512.971474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13513.067836] alg: No test for adler32 (adler32-zlib) [13514.373738] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [13514.677524] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [13516.375615] Key type lgssc registered [13518.071933] Lustre: Echo OBD driver; http://www.lustre.org/ [13565.422252] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [13581.334593] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13581.353336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13582.571546] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13582.621682] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13582.714236] Lustre: lustre-MDT0000: new disk, initializing [13582.833324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13582.846944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13587.675712] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [13607.087529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13607.506718] Lustre: lustre-OST0000: new disk, initializing [13607.513616] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13607.520478] Lustre: Skipped 1 previous similar message [13607.533379] Lustre: 437244:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13607.686233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13613.707717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13613.716184] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13613.749176] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13616.930181] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [13628.316104] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13647.826490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13647.992868] Lustre: 436224:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13648.115747] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13648.246194] Lustre: lustre-MDT0001: new disk, initializing [13648.440566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13648.481948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13648.497364] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13653.499787] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [13661.830632] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13662.079112] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13668.219831] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13668.506931] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13676.657505] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.40@tcp (stopping) [13679.593850] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13679.595254] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13679.612407] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13681.747842] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.40@tcp (stopping) [13681.757120] Lustre: Skipped 1 previous similar message [13684.707258] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13684.717974] Lustre: Skipped 1 previous similar message [13689.311365] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13689.489370] Lustre: server umount lustre-MDT0001 complete [13694.945714] LustreError: 437824:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13694.973193] LustreError: 437824:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [13700.067683] LustreError: 439589:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13700.109832] LustreError: 439589:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13705.188651] LustreError: 436234:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13705.223415] LustreError: 436234:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13710.311378] LustreError: 436235:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13710.314562] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13710.359390] Lustre: Skipped 1 previous similar message [13710.370771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13710.373183] Lustre: Skipped 1 previous similar message [13715.430948] LustreError: 436234:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13716.261271] Lustre: server umount lustre-OST0000 complete [13735.391128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13735.603316] Lustre: server umount lustre-MDT0000 complete [13754.718166] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [13758.630759] Key type lgssc unregistered [13759.117348] LNet: 440998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13759.122265] LNetError: 440998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13759.151660] LNet: Removed LNI 192.168.204.140@tcp [13759.993820] Key type .llcrypt unregistered [13759.995745] Key type ._llcrypt unregistered [13811.491397] Key type ._llcrypt registered [13811.494987] Key type .llcrypt registered [13811.628508] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [13811.999312] Key type .llcrypt unregistered [13812.004568] Key type ._llcrypt unregistered [13833.469115] Key type ._llcrypt registered [13833.473173] Key type .llcrypt registered [13833.573161] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [13834.954182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13835.221513] alg: No test for adler32 (adler32-zlib) [13836.242924] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [13836.547218] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [13838.247195] Key type lgssc registered [13839.896887] Lustre: Echo OBD driver; http://www.lustre.org/ [13874.904446] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13874.958655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13876.251186] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13876.309212] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13876.385843] Lustre: lustre-MDT0000: new disk, initializing [13876.473270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13876.493735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13882.343558] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [13893.506690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13893.595536] Lustre: 444382:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13893.654671] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13893.658330] Lustre: Skipped 1 previous similar message [13893.722630] Lustre: lustre-MDT0001: new disk, initializing [13893.781914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13893.823065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13893.834173] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13900.100981] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [13914.207372] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13922.082077] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13930.660410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13931.031864] Lustre: lustre-OST0000: new disk, initializing [13931.038260] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13931.055192] Lustre: 446362:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13931.191654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13937.000575] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [13940.845561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13940.854073] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13940.979803] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13947.234660] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13955.552666] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13955.562412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13955.594739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13956.583053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13956.590852] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13956.617153] Lustre: Skipped 1 previous similar message [13961.789712] Lustre: server umount lustre-MDT0000 complete [13966.024489] LustreError: 444372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454941 with bad export cookie 14132284903664370747 [13966.025709] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13966.039075] LustreError: 444372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13966.552566] Lustre: server umount lustre-MDT0001 complete [13981.413167] Lustre: server umount lustre-OST0000 complete [13992.795128] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 13:02:47 (1788454967) [14000.790318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14001.139829] LustreError: 448389:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14001.154601] LustreError: 448389:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [14001.283686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14005.575859] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [14006.753577] LustreError: 448390:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14010.851918] LustreError: 448389:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14014.638496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14015.211562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14020.659835] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [14030.009462] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14037.718861] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14057.265601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14057.512684] Lustre: Found index 0 for lustre-OST0000, updating log [14057.672057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14064.065437] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [14071.936667] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14072.059923] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14077.411439] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14077.421349] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14077.431763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14077.439167] Lustre: Skipped 1 previous similar message [14077.443869] Lustre: Skipped 2 previous similar messages [14079.342165] Lustre: server umount lustre-OST0000 complete [14095.272273] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14095.298209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14095.306137] Lustre: Skipped 4 previous similar messages [14100.451526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14100.473963] Lustre: Skipped 1 previous similar message [14101.700954] Lustre: server umount lustre-MDT0000 complete [14104.549233] LustreError: 449118:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14104.588988] LustreError: 449118:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14108.520424] LustreError: 448370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455084 with bad export cookie 14132284903664372147 [14108.536389] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14108.558840] LustreError: 448370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14109.378779] Lustre: server umount lustre-MDT0001 complete [14120.742903] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 13:04:54 (1788455094) [14124.383527] LNet: 1 local NIs in recovery (showing 1): 192.168.204.140@tcp [14750.952842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14751.522854] LustreError: 454805:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14751.550864] LustreError: 454805:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14751.601759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14756.406559] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [14756.835891] LustreError: 454806:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14761.954500] LustreError: 454805:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14765.460650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14765.838893] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14770.960641] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [14780.672519] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14788.653808] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14803.359859] LNet: 1 local NIs in recovery (showing 1): 192.168.204.140@tcp [14808.815172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14809.044220] Lustre: Found index 0 for lustre-OST0000, updating log [14809.268858] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14815.241710] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [14822.783461] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14826.027327] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14826.045294] Lustre: Skipped 1 previous similar message [14826.050842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14839.263147] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14839.432387] Lustre: server umount lustre-OST0000 complete [14844.079486] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14844.088817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14844.101223] Lustre: Skipped 1 previous similar message [14844.112216] Lustre: Skipped 1 previous similar message [14846.177401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14846.194310] Lustre: Skipped 3 previous similar messages [14849.426593] Lustre: server umount lustre-MDT0000 complete [14850.273862] LustreError: 455801:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14850.285913] LustreError: 455801:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14853.732193] LustreError: 454785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455829 with bad export cookie 14132284903664373225 [14853.753273] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14854.153715] Lustre: server umount lustre-MDT0001 complete [14862.752240] LNet: 1 local NIs in recovery (showing 1): 192.168.204.140@tcp [14888.437746] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [14901.275528] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [14936.954653] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 13:18:31 (1788455911) [14954.990328] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [14957.476231] Key type lgssc unregistered [14957.838341] LNet: 461430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14957.848693] LNetError: 461430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15070.503846] LNet: Removed LNI 192.168.204.140@tcp [15071.229646] Key type .llcrypt unregistered [15071.231130] Key type ._llcrypt unregistered [15090.407505] Key type ._llcrypt registered [15090.410122] Key type .llcrypt registered [15090.551965] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [15091.691809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15091.918417] alg: No test for adler32 (adler32-zlib) [15093.107493] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [15093.321606] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [15095.119172] Key type lgssc registered [15096.297596] Lustre: Echo OBD driver; http://www.lustre.org/ [15101.477368] LNet: Added LNI 192.168.204.140@tcp100 [8/256/0/180] [15102.202923] LNet: Added LNI 192.168.204.140@tcp101 [8/256/0/180] [15103.048391] LNet: Added LNI 192.168.204.140@tcp102 [8/256/0/180] [15103.750145] LNet: Added LNI 192.168.204.140@tcp103 [8/256/0/180] [15104.427913] LNet: Added LNI 192.168.204.140@tcp104 [8/256/0/180] [15105.213109] LNet: Added LNI 192.168.204.140@tcp105 [8/256/0/180] [15106.062805] LNet: Added LNI 192.168.204.140@tcp106 [8/256/0/180] [15107.186503] LNet: Added LNI 192.168.204.140@tcp107 [8/256/0/180] [15108.288391] LNet: Added LNI 192.168.204.140@tcp108 [8/256/0/180] [15109.613344] LNet: Added LNI 192.168.204.140@tcp109 [8/256/0/180] [15110.651235] LNet: Added LNI 192.168.204.140@tcp110 [8/256/0/180] [15111.981912] LNet: Added LNI 192.168.204.140@tcp111 [8/256/0/180] [15113.362569] LNet: Added LNI 192.168.204.140@tcp112 [8/256/0/180] [15114.363757] LNet: Added LNI 192.168.204.140@tcp113 [8/256/0/180] [15115.359405] LNet: Added LNI 192.168.204.140@tcp114 [8/256/0/180] [15116.226577] LNet: Added LNI 192.168.204.140@tcp115 [8/256/0/180] [15117.296668] LNet: Added LNI 192.168.204.140@tcp116 [8/256/0/180] [15118.250805] LNet: Added LNI 192.168.204.140@tcp117 [8/256/0/180] [15119.763726] LNet: Added LNI 192.168.204.140@tcp118 [8/256/0/180] [15120.875703] LNet: Added LNI 192.168.204.140@tcp119 [8/256/0/180] [15121.908676] LNet: Added LNI 192.168.204.140@tcp120 [8/256/0/180] [15122.902402] LNet: Added LNI 192.168.204.140@tcp121 [8/256/0/180] [15123.997171] LNet: Added LNI 192.168.204.140@tcp122 [8/256/0/180] [15125.583563] LNet: Added LNI 192.168.204.140@tcp123 [8/256/0/180] [15127.136218] LNet: Added LNI 192.168.204.140@tcp124 [8/256/0/180] [15128.659907] LNet: Added LNI 192.168.204.140@tcp125 [8/256/0/180] [15129.864855] LNet: Added LNI 192.168.204.140@tcp126 [8/256/0/180] [15131.245741] LNet: Added LNI 192.168.204.140@tcp127 [8/256/0/180] [15132.593657] LNet: Added LNI 192.168.204.140@tcp128 [8/256/0/180] [15133.867752] LNet: Added LNI 192.168.204.140@tcp129 [8/256/0/180] [15135.165568] LNet: Added LNI 192.168.204.140@tcp130 [8/256/0/180] [15136.524881] LNet: Added LNI 192.168.204.140@tcp131 [8/256/0/180] [15138.004392] LNet: Added LNI 192.168.204.140@tcp132 [8/256/0/180] [15139.674957] LNet: Added LNI 192.168.204.140@tcp133 [8/256/0/180] [15140.669671] LNet: Added LNI 192.168.204.140@tcp134 [8/256/0/180] [15141.899769] LNet: Added LNI 192.168.204.140@tcp135 [8/256/0/180] [15150.660098] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15150.718119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15151.909191] Lustre: 464463:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.140@tcp: can't add NID 192.168.204.140@tcp130: rc = -75 [15151.997110] LNetError: 464483:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.140 [15152.008545] LNetError: Protocol error connecting to 192.168.204.140@tcp121 at host 192.168.204.140:988: is it running a compatible version of Lustre? [15152.027412] LNetError: Skipped 11 previous similar messages [15152.075151] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15152.195415] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15152.322887] Lustre: lustre-MDT0000: new disk, initializing [15152.420272] Lustre: 464521:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.140@tcp: can't add NID 192.168.204.140@tcp130: rc = -75 [15152.436931] Lustre: 464521:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15152.469578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15152.503558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15152.740876] LNet: 6 local NIs in recovery (showing 6): 192.168.204.140@tcp125, 192.168.204.140@tcp126, 192.168.204.140@tcp127, 192.168.204.140@tcp129, 192.168.204.140@tcp128, 192.168.204.140@tcp121 [15153.760697] LNetError: 464488:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.140 [15153.780663] LNetError: 464488:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 13 previous similar messages [15153.804396] LNetError: Protocol error connecting to 192.168.204.140@tcp126 at host 192.168.204.140:988: is it running a compatible version of Lustre? [15155.818051] LNetError: 464497:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.140 [15155.827596] LNetError: Protocol error connecting to 192.168.204.140@tcp126 at host 192.168.204.140:988: is it running a compatible version of Lustre? [15156.806935] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [15159.920555] LNetError: 464484:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.140 [15159.936529] LNetError: Protocol error connecting to 192.168.204.140@tcp126 at host 192.168.204.140:988: is it running a compatible version of Lustre? [15167.035234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15167.075258] LNetError: 464489:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.140 [15167.094958] LNetError: Protocol error connecting to 192.168.204.140@tcp126 at host 192.168.204.140:988: is it running a compatible version of Lustre? [15167.210192] Lustre: 464478:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15167.316390] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15167.324584] Lustre: Skipped 1 previous similar message [15167.436778] Lustre: lustre-MDT0001: new disk, initializing [15167.491763] Lustre: 465284:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.140@tcp: can't add NID 192.168.204.140@tcp130: rc = -75 [15167.498583] Lustre: 465284:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15167.672740] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15167.732890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15167.752949] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15172.879565] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [15182.465444] LNetError: 464496:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.140 [15182.481886] LNetError: Protocol error connecting to 192.168.204.140@tcp124 at host 192.168.204.140:988: is it running a compatible version of Lustre? [15186.410769] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15194.651732] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15198.881143] LNetError: 464496:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.140 [15198.894807] LNetError: 464496:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 11 previous similar messages [15198.911168] LNetError: Protocol error connecting to 192.168.204.140@tcp122 at host 192.168.204.140:988: is it running a compatible version of Lustre? [15198.933475] LNetError: Skipped 11 previous similar messages [15202.904942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15203.457474] Lustre: lustre-OST0000: new disk, initializing [15203.467354] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15203.479062] Lustre: 466458:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15203.635938] Lustre: 466463:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.140@tcp: can't add NID 192.168.204.140@tcp130: rc = -75 [15203.648512] Lustre: 466463:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [15203.739486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15211.456490] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [15212.085282] Lustre: 466835:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.140@tcp: can't add NID 192.168.204.140@tcp130: rc = -75 [15212.109919] Lustre: 466835:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15212.237692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15212.254188] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15212.469291] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15213.151593] LNet: 1 local NIs in recovery (showing 1): 192.168.204.140@tcp126 [15222.621612] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15233.003320] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15233.011525] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15233.026315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15233.041208] Lustre: Skipped 1 previous similar message [15234.726893] LNetError: 464498:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.140 [15234.743157] LNetError: 464498:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 17 previous similar messages [15234.755461] LNetError: Protocol error connecting to 192.168.204.140@tcp111 at host 192.168.204.140:988: is it running a compatible version of Lustre? [15234.772067] LNetError: Skipped 17 previous similar messages [15238.125649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15238.142034] Lustre: Skipped 1 previous similar message [15243.240744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15246.303197] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15246.502234] Lustre: server umount lustre-OST0000 complete [15253.474630] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15253.479136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15253.494058] Lustre: Skipped 2 previous similar messages [15253.507734] Lustre: Skipped 2 previous similar messages [15258.595076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15258.601473] Lustre: Skipped 1 previous similar message [15265.247372] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15265.485625] Lustre: server umount lustre-MDT0000 complete [15270.238562] LustreError: 464473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788456245 with bad export cookie 14045802279142535124 [15270.240923] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15270.257542] LustreError: 464473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15271.039368] Lustre: server umount lustre-MDT0001 complete [15283.419937] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [15286.894625] Key type lgssc unregistered [15287.609318] LNet: 468378:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15287.644134] LNetError: 468378:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15288.671349] LNet: Removed LNI 192.168.204.140@tcp [15288.680307] LNetError: 468378:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15289.695227] LNet: Removed LNI 192.168.204.140@tcp100 [15289.697068] LNetError: 468378:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15289.712550] LNet: Removed LNI 192.168.204.140@tcp101 [15290.719162] LNet: Removed LNI 192.168.204.140@tcp102 [15291.743311] LNet: Removed LNI 192.168.204.140@tcp103 [15291.746609] LNetError: 468378:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15291.759270] LNetError: 468378:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15291.767901] LNet: Removed LNI 192.168.204.140@tcp104 [15292.831727] LNet: Removed LNI 192.168.204.140@tcp105 [15293.855184] LNet: Removed LNI 192.168.204.140@tcp106 [15294.879263] LNet: Removed LNI 192.168.204.140@tcp107 [15295.903178] LNet: Removed LNI 192.168.204.140@tcp108 [15295.909589] LNetError: 468378:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15295.925994] LNetError: 468378:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15295.941606] LNet: Removed LNI 192.168.204.140@tcp109 [15296.994101] LNet: Removed LNI 192.168.204.140@tcp110 [15298.015182] LNet: Removed LNI 192.168.204.140@tcp111 [15299.039686] LNet: Removed LNI 192.168.204.140@tcp112 [15300.063518] LNet: Removed LNI 192.168.204.140@tcp113 [15301.087138] LNet: Removed LNI 192.168.204.140@tcp114 [15302.112257] LNet: Removed LNI 192.168.204.140@tcp115 [15303.135370] LNet: Removed LNI 192.168.204.140@tcp116 [15304.161520] LNet: Removed LNI 192.168.204.140@tcp117 [15304.172553] LNetError: 468378:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15304.181240] LNetError: 468378:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15304.193394] LNet: Removed LNI 192.168.204.140@tcp118 [15305.247938] LNet: Removed LNI 192.168.204.140@tcp119 [15306.271179] LNet: Removed LNI 192.168.204.140@tcp120 [15307.295142] LNet: Removed LNI 192.168.204.140@tcp121 [15308.319176] LNet: Removed LNI 192.168.204.140@tcp122 [15309.343241] LNet: Removed LNI 192.168.204.140@tcp123 [15310.367314] LNet: Removed LNI 192.168.204.140@tcp124 [15311.391161] LNet: Removed LNI 192.168.204.140@tcp125 [15311.394544] LNet: Removed LNI 192.168.204.140@tcp126 [15312.415307] LNet: Removed LNI 192.168.204.140@tcp127 [15313.439138] LNet: Removed LNI 192.168.204.140@tcp128 [15314.463154] LNet: Removed LNI 192.168.204.140@tcp129 [15314.466773] LNet: Removed LNI 192.168.204.140@tcp130 [15314.486657] LNet: Removed LNI 192.168.204.140@tcp131 [15314.489585] LNet: Removed LNI 192.168.204.140@tcp132 [15314.500906] LNet: Removed LNI 192.168.204.140@tcp133 [15314.510379] LNet: Removed LNI 192.168.204.140@tcp134 [15314.541844] LNet: Removed LNI 192.168.204.140@tcp135 [15315.682446] Key type .llcrypt unregistered [15315.691637] Key type ._llcrypt unregistered [15342.098407] Key type ._llcrypt registered [15342.100300] Key type .llcrypt registered [15342.256139] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [15360.014659] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [15360.801630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15360.870324] alg: No test for adler32 (adler32-zlib) [15362.043963] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [15362.308634] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [15363.992467] Key type lgssc registered [15365.152740] Lustre: Echo OBD driver; http://www.lustre.org/ [15412.462612] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 13:26:26 (1788456386) [15434.502673] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [15438.104837] Key type lgssc unregistered [15438.510542] LNet: 472491:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15438.515045] LNetError: 472491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15438.532137] LNet: Removed LNI 192.168.204.140@tcp [15439.443683] Key type .llcrypt unregistered [15439.447065] Key type ._llcrypt unregistered [15460.512511] Key type ._llcrypt registered [15460.514788] Key type .llcrypt registered [15460.659683] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [15461.996587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15462.015163] alg: No test for adler32 (adler32-zlib) [15463.097569] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [15463.338729] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [15465.087338] Key type lgssc registered [15466.213646] Lustre: Echo OBD driver; http://www.lustre.org/ [15477.677133] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15477.750388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15479.110133] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15479.134872] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15479.241346] Lustre: lustre-MDT0000: new disk, initializing [15479.324806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15479.359492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15484.796641] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [15496.114207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15496.260767] Lustre: 473706:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15496.308516] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15496.313066] Lustre: Skipped 1 previous similar message [15496.383613] Lustre: lustre-MDT0001: new disk, initializing [15496.505770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15496.531474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15496.536181] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15500.613557] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [15514.653086] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15523.249427] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15532.639434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15532.901480] Lustre: lustre-OST0000: new disk, initializing [15532.910256] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15532.927591] Lustre: 475677:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15533.067422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15541.466216] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [15542.892269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15542.903131] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15543.029808] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15553.591245] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15558.898966] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing lnet_if_list [15566.679418] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15566.734328] LNet: Added LNI 192.168.204.140@tcp10 [8/256/0/180] [15573.401785] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15573.453064] LNet: Added LNI 192.168.204.140@tcp11 [8/256/0/180] [15579.128797] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15579.153489] LNet: Added LNI 192.168.204.140@tcp12 [8/256/0/180] [15584.932520] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15584.980342] LNet: Added LNI 192.168.204.140@tcp13 [8/256/0/180] [15590.281271] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15590.336058] LNet: Added LNI 192.168.204.140@tcp14 [8/256/0/180] [15595.858510] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15595.897394] LNet: Added LNI 192.168.204.140@tcp15 [8/256/0/180] [15602.736817] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15602.763037] LNet: Added LNI 192.168.204.140@tcp16 [8/256/0/180] [15607.948108] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15607.969122] LNet: Added LNI 192.168.204.140@tcp17 [8/256/0/180] [15613.727803] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15613.754980] LNet: Added LNI 192.168.204.140@tcp18 [8/256/0/180] [15620.635555] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15620.677289] LNet: Added LNI 192.168.204.140@tcp19 [8/256/0/180] [15629.167391] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15629.247563] LNet: Added LNI 192.168.204.140@tcp20 [8/256/0/180] [15638.036397] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15638.092077] LNet: Added LNI 192.168.204.140@tcp21 [8/256/0/180] [15645.681340] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15645.737671] LNet: Added LNI 192.168.204.140@tcp22 [8/256/0/180] [15653.498714] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15653.523158] LNet: Added LNI 192.168.204.140@tcp23 [8/256/0/180] [15661.599550] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15661.650048] LNet: Added LNI 192.168.204.140@tcp24 [8/256/0/180] [15668.286039] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15668.340810] LNet: Added LNI 192.168.204.140@tcp25 [8/256/0/180] [15678.203314] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15678.241538] LNet: Added LNI 192.168.204.140@tcp26 [8/256/0/180] [15685.597296] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15685.635440] LNet: Added LNI 192.168.204.140@tcp27 [8/256/0/180] [15692.551375] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15692.584596] LNet: Added LNI 192.168.204.140@tcp28 [8/256/0/180] [15699.757951] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15699.786632] LNet: Added LNI 192.168.204.140@tcp29 [8/256/0/180] [15706.915884] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15706.947123] LNet: Added LNI 192.168.204.140@tcp30 [8/256/0/180] [15714.104435] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15714.125586] LNet: Added LNI 192.168.204.140@tcp31 [8/256/0/180] [15722.706324] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15722.744883] LNet: Added LNI 192.168.204.140@tcp32 [8/256/0/180] [15732.211918] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15732.242947] LNet: Added LNI 192.168.204.140@tcp33 [8/256/0/180] [15741.486658] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15741.510874] LNet: Added LNI 192.168.204.140@tcp34 [8/256/0/180] [15749.390783] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15749.438377] LNet: Added LNI 192.168.204.140@tcp35 [8/256/0/180] [15758.066973] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15758.127642] LNet: Added LNI 192.168.204.140@tcp36 [8/256/0/180] [15765.812984] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15765.866437] LNet: Added LNI 192.168.204.140@tcp37 [8/256/0/180] [15774.233757] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15774.273547] LNet: Added LNI 192.168.204.140@tcp38 [8/256/0/180] [15783.887285] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15783.992042] LNet: Added LNI 192.168.204.140@tcp39 [8/256/0/180] [15793.263376] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15793.284503] LNet: Added LNI 192.168.204.140@tcp40 [8/256/0/180] [15803.488837] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15803.509725] LNet: Added LNI 192.168.204.140@tcp41 [8/256/0/180] [15814.185148] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15814.249884] LNet: Added LNI 192.168.204.140@tcp42 [8/256/0/180] [15825.143292] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15825.167812] LNet: Added LNI 192.168.204.140@tcp43 [8/256/0/180] [15834.400621] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15834.452972] LNet: Added LNI 192.168.204.140@tcp44 [8/256/0/180] [15843.726949] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15843.790962] LNet: Added LNI 192.168.204.140@tcp45 [8/256/0/180] [15853.225276] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15853.272538] LNet: Added LNI 192.168.204.140@tcp46 [8/256/0/180] [15861.981304] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15862.019726] LNet: Added LNI 192.168.204.140@tcp47 [8/256/0/180] [15870.116211] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15870.184733] LNet: Added LNI 192.168.204.140@tcp48 [8/256/0/180] [15878.561575] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15878.588017] LNet: Added LNI 192.168.204.140@tcp49 [8/256/0/180] [15887.948931] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15888.009061] LNet: Added LNI 192.168.204.140@tcp50 [8/256/0/180] [15916.512561] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15916.540392] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15916.579793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15919.724794] Lustre: server umount lustre-MDT0000 complete [15923.885131] LustreError: 473699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788456899 with bad export cookie 869699234614551091 [15923.888540] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15923.898844] LustreError: 473699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15924.281298] Lustre: server umount lustre-MDT0001 complete [15942.623320] Lustre: 472981:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788456902/real 1788456902] req@00000000251ecf4e x1875332498120320/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788456918 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15942.685299] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15942.701496] Lustre: Skipped 2 previous similar messages [15943.647158] Lustre: 472980:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788456903/real 1788456903] req@000000008bd83a8a x1875332498120448/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788456919 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15947.744270] Lustre: 472978:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788456907/real 1788456907] req@00000000670ffade x1875332498120576/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788456923 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15947.761283] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [15949.858323] Lustre: 472978:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788456909/real 1788456909] req@00000000172e5d8f x1875332498120832/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788456925 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15954.911347] Lustre: 472980:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788456914/real 1788456914] req@00000000dfe5bf34 x1875332498120960/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788456930 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15958.377754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15969.313488] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xc11cad894a12cc8 [15969.340077] Lustre: MGC192.168.204.140@tcp: Connection restored to 0@lo (at 0@lo) [15969.705445] LustreError: 475671:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15969.739994] LustreError: 475671:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [15969.823848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15971.874157] LustreError: 475672:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15971.883495] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15973.943420] LustreError: 475671:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15976.164464] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [15978.989539] LustreError: 475672:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15984.099630] LustreError: 475671:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15988.466909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15989.312204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15994.352407] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15994.884423] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [16004.358649] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16011.690833] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16036.320287] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16036.341750] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16036.369760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16036.381217] Lustre: Skipped 3 previous similar messages [16040.430871] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16040.435206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16040.438980] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16040.484968] Lustre: Skipped 1 previous similar message [16041.815542] Lustre: server umount lustre-OST0000 complete [16050.656519] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16050.661670] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16050.672732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16050.682302] Lustre: Skipped 1 previous similar message [16050.689015] Lustre: Skipped 1 previous similar message [16052.599979] Lustre: server umount lustre-MDT0000 complete [16055.780313] LustreError: 499933:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16055.802932] LustreError: 499933:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [16056.869952] LustreError: 473700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788457032 with bad export cookie 869699234614553800 [16056.875261] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16056.891027] LustreError: 473700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16057.600853] Lustre: server umount lustre-MDT0001 complete [16072.423547] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [16076.044829] Key type lgssc unregistered [16076.488259] LNet: 502734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16076.495774] LNetError: 502734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16076.500939] LNet: Removed LNI 192.168.204.140@tcp [16076.504055] LNet: Removed LNI 192.168.204.140@tcp10 [16076.506953] LNet: Removed LNI 192.168.204.140@tcp11 [16076.512443] LNet: Removed LNI 192.168.204.140@tcp12 [16076.520067] LNet: Removed LNI 192.168.204.140@tcp13 [16076.530898] LNet: Removed LNI 192.168.204.140@tcp14 [16076.535835] LNet: Removed LNI 192.168.204.140@tcp15 [16076.541556] LNet: Removed LNI 192.168.204.140@tcp16 [16076.545359] LNet: Removed LNI 192.168.204.140@tcp17 [16076.554778] LNet: Removed LNI 192.168.204.140@tcp18 [16076.560945] LNet: Removed LNI 192.168.204.140@tcp19 [16076.564983] LNet: Removed LNI 192.168.204.140@tcp20 [16076.568442] LNet: Removed LNI 192.168.204.140@tcp21 [16076.575983] LNet: Removed LNI 192.168.204.140@tcp22 [16076.584306] LNet: Removed LNI 192.168.204.140@tcp23 [16076.586336] LNet: Removed LNI 192.168.204.140@tcp24 [16076.598355] LNet: Removed LNI 192.168.204.140@tcp25 [16076.602954] LNet: Removed LNI 192.168.204.140@tcp26 [16076.610346] LNet: Removed LNI 192.168.204.140@tcp27 [16076.621402] LNet: Removed LNI 192.168.204.140@tcp28 [16076.634785] LNet: Removed LNI 192.168.204.140@tcp29 [16076.639560] LNet: Removed LNI 192.168.204.140@tcp30 [16076.655278] LNet: Removed LNI 192.168.204.140@tcp31 [16076.659802] LNet: Removed LNI 192.168.204.140@tcp32 [16076.672818] LNet: Removed LNI 192.168.204.140@tcp33 [16076.677570] LNet: Removed LNI 192.168.204.140@tcp34 [16076.681386] LNet: Removed LNI 192.168.204.140@tcp35 [16076.686602] LNet: Removed LNI 192.168.204.140@tcp36 [16076.690877] LNet: Removed LNI 192.168.204.140@tcp37 [16076.694905] LNet: Removed LNI 192.168.204.140@tcp38 [16076.698935] LNet: Removed LNI 192.168.204.140@tcp39 [16076.704382] LNet: Removed LNI 192.168.204.140@tcp40 [16076.713796] LNet: Removed LNI 192.168.204.140@tcp41 [16076.736254] LNet: Removed LNI 192.168.204.140@tcp42 [16076.741302] LNet: Removed LNI 192.168.204.140@tcp43 [16076.747758] LNet: Removed LNI 192.168.204.140@tcp44 [16076.754891] LNet: Removed LNI 192.168.204.140@tcp45 [16076.760632] LNet: Removed LNI 192.168.204.140@tcp46 [16076.775787] LNet: Removed LNI 192.168.204.140@tcp47 [16076.782914] LNet: Removed LNI 192.168.204.140@tcp48 [16076.790384] LNet: Removed LNI 192.168.204.140@tcp49 [16076.810315] LNet: Removed LNI 192.168.204.140@tcp50 [16077.746597] Key type .llcrypt unregistered [16077.749641] Key type ._llcrypt unregistered [16103.435225] Key type ._llcrypt registered [16103.437262] Key type .llcrypt registered [16103.594917] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [16123.567100] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [16125.309744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16125.344853] alg: No test for adler32 (adler32-zlib) [16126.518958] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [16126.837305] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [16128.568660] Key type lgssc registered [16129.878337] Lustre: Echo OBD driver; http://www.lustre.org/ [16169.605525] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16169.657159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16170.997276] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16171.060965] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16171.176711] Lustre: lustre-MDT0000: new disk, initializing [16171.348864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16171.375767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16176.401573] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [16187.926089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16188.047821] Lustre: 505937:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16188.112962] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16188.117377] Lustre: Skipped 1 previous similar message [16188.181130] Lustre: lustre-MDT0001: new disk, initializing [16188.277665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16188.332184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16188.347577] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16193.439336] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [16204.497785] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16211.080610] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16220.728633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16220.974274] Lustre: *** cfs_fail_loc=90f, val=0*** [16221.048969] Lustre: lustre-OST0000: new disk, initializing [16221.053896] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16221.064474] Lustre: 507952:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16221.187156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16228.604626] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [16230.009093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16230.024349] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16230.354483] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16241.126506] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16250.850823] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16250.861255] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16250.877958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16250.898204] Lustre: Skipped 1 previous similar message [16255.970073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16255.996572] Lustre: Skipped 1 previous similar message [16256.841940] Lustre: server umount lustre-OST0000 complete [16265.184345] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16265.190196] LustreError: Skipped 1 previous similar message [16265.200851] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16265.222651] Lustre: Skipped 1 previous similar message [16265.235520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16266.210752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16268.220528] Lustre: server umount lustre-MDT0000 complete [16272.618608] LustreError: 506731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788457248 with bad export cookie 9886094214993992099 [16272.622656] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16272.626320] LustreError: 506731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16273.209886] Lustre: server umount lustre-MDT0001 complete [16283.955995] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [16286.823423] Key type lgssc unregistered [16287.124099] LNet: 509863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16287.128523] LNetError: 509863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16288.176165] LNet: Removed LNI 192.168.204.140@tcp [16289.368110] Key type .llcrypt unregistered [16289.370364] Key type ._llcrypt unregistered [16313.394461] Key type ._llcrypt registered [16313.398757] Key type .llcrypt registered [16313.497666] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [16314.780439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16314.819743] alg: No test for adler32 (adler32-zlib) [16316.080917] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [16316.392831] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [16318.119234] Key type lgssc registered [16319.361454] Lustre: Echo OBD driver; http://www.lustre.org/ [16328.376585] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16328.403669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16329.763263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16334.532869] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [16339.943780] LustreError: 511136:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16339.964220] LustreError: 511136:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16342.964003] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16343.369990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16347.565191] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [16359.708971] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16368.021771] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16376.063986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16376.545357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16384.402756] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [16394.126722] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16411.874485] loop0: detected capacity change from 0 to 204800000 [16412.090758] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16412.108379] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16412.119775] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16418.886669] loop0: detected capacity change from 0 to 204800000 [16419.080683] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16419.091828] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16419.110852] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16425.356840] loop0: detected capacity change from 0 to 204800000 [16431.437336] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16431.643942] Lustre: fs15246-OST0000: new disk, initializing [16431.649648] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16431.660068] Lustre: 514754:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16431.733952] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16437.605390] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [16448.615888] loop1: detected capacity change from 0 to 204800000 [16454.817927] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16455.033177] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16455.103499] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16455.197913] Lustre: fs15246-MDT0000: new disk, initializing [16455.435101] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16455.470192] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16455.609427] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16456.685178] Lustre: 515880:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000a9cd859b x1875333393467776/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788457448 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16461.389192] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [16465.795261] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16465.811900] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16465.864276] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16490.471756] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16490.496589] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16495.596410] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16496.041784] Lustre: server umount fs15246-MDT0000 complete [16500.709696] LustreError: 511131:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16505.844560] LustreError: 511133:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16506.492836] Lustre: server umount fs15246-OST0000 complete [16521.187858] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16521.189495] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16521.197082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16521.204521] Lustre: Skipped 1 previous similar message [16526.309212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16526.316093] Lustre: Skipped 2 previous similar messages [16527.336608] Lustre: server umount lustre-OST0000 complete [16531.424361] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16531.433316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16531.444819] Lustre: Skipped 1 previous similar message [16537.359425] Lustre: server umount lustre-MDT0000 complete [16541.664992] LustreError: 511132:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16541.686314] LustreError: 511132:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16542.038288] LustreError: 511117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788457517 with bad export cookie 1105878143446326037 [16542.040926] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16542.054431] LustreError: 511117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16542.408224] Lustre: server umount lustre-MDT0001 complete [16553.576646] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [16556.623182] Key type lgssc unregistered [16557.076413] LNet: 518981:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16557.089955] LNetError: 518981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16557.128173] LNet: Removed LNI 192.168.204.140@tcp [16558.264452] Key type .llcrypt unregistered [16558.273793] Key type ._llcrypt unregistered [16580.975408] Key type ._llcrypt registered [16580.977707] Key type .llcrypt registered [16581.081156] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [16582.178499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16582.197439] alg: No test for adler32 (adler32-zlib) [16583.226239] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [16583.438353] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [16585.111191] Key type lgssc registered [16586.180908] Lustre: Echo OBD driver; http://www.lustre.org/ [16633.954434] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [16645.011069] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [16686.424295] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16686.524275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16687.901706] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16687.975165] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16688.144947] Lustre: lustre-MDT0000: new disk, initializing [16688.307383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16688.348522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16694.680214] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [16706.661052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16706.773683] Lustre: 523449:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16706.890199] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16706.903414] Lustre: Skipped 1 previous similar message [16706.971633] Lustre: lustre-MDT0001: new disk, initializing [16707.191633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16707.271911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16707.294844] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16711.830217] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [16723.924910] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16731.213925] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16742.109831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16742.830963] Lustre: lustre-OST0000: new disk, initializing [16742.837599] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16742.846232] Lustre: 525421:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16742.981836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16752.744689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16752.757406] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16752.926463] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16753.897855] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [16765.860702] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16771.268981] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16771.504962] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16776.816298] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16777.031795] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16783.851086] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16783.854477] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16783.885969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16783.900098] Lustre: Skipped 1 previous similar message [16786.027932] Lustre: server umount lustre-OST0000 complete [16794.092715] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16794.100833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16794.107483] Lustre: Skipped 2 previous similar messages [16794.110445] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16794.117485] Lustre: Skipped 1 previous similar message [16794.123806] LustreError: Skipped 1 previous similar message [16796.359262] Lustre: server umount lustre-MDT0000 complete [16801.165984] LustreError: 523441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788457776 with bad export cookie 2306716906965654200 [16801.167536] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16801.193210] LustreError: 523441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16801.895402] Lustre: server umount lustre-MDT0001 complete [16838.356608] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [16852.468482] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [16897.303768] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16589 sec ========= 13:51:11 (1788457871) [16899.563973] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:51:13 (1788457873) === [16904.324615] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:51:18 (1788457878) === [16950.014968] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [16954.067601] Key type lgssc unregistered [16954.467272] LNet: 532095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16954.475019] LNetError: 532095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16954.505169] LNet: Removed LNI 192.168.204.140@tcp [16956.133467] Key type .llcrypt unregistered [16956.135663] Key type ._llcrypt unregistered