[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 764135339 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003090] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005019] kvm-guest: setup PV IPIs [ 0.008552] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009029] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010019] pid_max: default: 32768 minimum: 301 [ 0.011171] LSM: Security Framework initializing [ 0.012065] Yama: becoming mindful. [ 0.013044] SELinux: Initializing. [ 0.014074] *** VALIDATE selinux *** [ 0.023364] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029272] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.031172] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032120] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033142] *** VALIDATE tmpfs *** [ 0.035393] *** VALIDATE proc *** [ 0.036378] *** VALIDATE cgroup *** [ 0.038006] *** VALIDATE cgroup2 *** [ 0.039377] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.041172] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.042011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.043038] Spectre V2 : User space: Vulnerable [ 0.044010] Speculative Store Bypass: Vulnerable [ 0.048050] debug: unmapping init [mem 0xffffffffb9659000-0xffffffffb9660fff] [ 0.051150] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.052878] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.053027] ... version: 2 [ 0.054017] ... bit width: 48 [ 0.055016] ... generic registers: 4 [ 0.056015] ... value mask: 0000ffffffffffff [ 0.057020] ... max period: 00007fffffffffff [ 0.058020] ... fixed-purpose events: 3 [ 0.059015] ... event mask: 000000070000000f [ 0.060345] rcu: Hierarchical SRCU implementation. [ 0.063051] smp: Bringing up secondary CPUs ... [ 0.064821] x86: Booting SMP configuration: [ 0.065041] .... node #0, CPUs: #1 #2 #3 [ 0.069428] smp: Brought up 1 node, 4 CPUs [ 0.071048] smpboot: Max logical packages: 1 [ 0.072014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.193049] node 0 deferred pages initialised in 116ms [ 0.202594] devtmpfs: initialized [ 0.205585] x86/mm: Memory block size: 128MB [ 0.210983] gcov: version magic: 0x41383552 [ 0.215445] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.219286] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.221505] pinctrl core: initialized pinctrl subsystem [ 0.223204] [ 0.223829] ************************************************************* [ 0.226018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.228015] ** ** [ 0.230018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.232022] ** ** [ 0.236059] ** This means that this kernel is built to expose internal ** [ 0.239020] ** IOMMU data structures, which may compromise security on ** [ 0.242018] ** your system. ** [ 0.244014] ** ** [ 0.246013] ** If you see this message and you are not debugging the ** [ 0.247063] ** kernel, report this immediately to your vendor! ** [ 0.249020] ** ** [ 0.251016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.254017] ************************************************************* [ 0.257089] NET: Registered protocol family 16 [ 0.259693] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.261092] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.264085] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.268142] cpuidle: using governor menu [ 0.269739] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.272772] PCI: Using configuration type 1 for base access [ 0.275250] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.285159] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.287049] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.291031] cryptd: max_cpu_qlen set to 1000 [ 0.294323] ACPI: Added _OSI(Module Device) [ 0.296021] ACPI: Added _OSI(Processor Device) [ 0.297017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.299015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.303000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.312722] ACPI: Interpreter enabled [ 0.313081] ACPI: PM: (supports S0 S3 S4 S5) [ 0.315014] ACPI: Using IOAPIC for interrupt routing [ 0.316152] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.320599] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.332356] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.334120] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.336024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.340112] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.344610] acpiphp: Slot [2] registered [ 0.346231] acpiphp: Slot [5] registered [ 0.348263] acpiphp: Slot [6] registered [ 0.349182] acpiphp: Slot [7] registered [ 0.350181] acpiphp: Slot [8] registered [ 0.352120] acpiphp: Slot [9] registered [ 0.353162] acpiphp: Slot [10] registered [ 0.355153] acpiphp: Slot [3] registered [ 0.356137] acpiphp: Slot [4] registered [ 0.357161] acpiphp: Slot [11] registered [ 0.359181] acpiphp: Slot [12] registered [ 0.360127] acpiphp: Slot [13] registered [ 0.361106] acpiphp: Slot [14] registered [ 0.363177] acpiphp: Slot [15] registered [ 0.364128] acpiphp: Slot [16] registered [ 0.365215] acpiphp: Slot [17] registered [ 0.367110] acpiphp: Slot [18] registered [ 0.368106] acpiphp: Slot [19] registered [ 0.370141] acpiphp: Slot [20] registered [ 0.371112] acpiphp: Slot [21] registered [ 0.372175] acpiphp: Slot [22] registered [ 0.374092] acpiphp: Slot [23] registered [ 0.375127] acpiphp: Slot [24] registered [ 0.376111] acpiphp: Slot [25] registered [ 0.378141] acpiphp: Slot [26] registered [ 0.380384] acpiphp: Slot [27] registered [ 0.382287] acpiphp: Slot [28] registered [ 0.384107] acpiphp: Slot [29] registered [ 0.386126] acpiphp: Slot [30] registered [ 0.387110] acpiphp: Slot [31] registered [ 0.389126] PCI host bridge to bus 0000:00 [ 0.390027] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.393032] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.395031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.398033] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.400103] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.403039] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.405401] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.409293] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.412526] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.422697] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.427455] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.430024] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.432020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.434026] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.437162] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.439815] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.442051] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.444843] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.448864] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.460019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.464021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.472183] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.480024] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.488025] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.531031] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.544979] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.555024] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.564021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.583025] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.594345] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.604020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.610029] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.629035] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.641477] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.650026] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.656022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.677028] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.691830] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.699069] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.704029] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.724026] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.735898] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.743023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.751025] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.772030] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.786177] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.790502] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.792471] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.794503] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.797241] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.803039] iommu: Default domain type: Passthrough [ 0.804494] SCSI subsystem initialized [ 0.806171] ACPI: bus type USB registered [ 0.807102] usbcore: registered new interface driver usbfs [ 0.809133] usbcore: registered new interface driver hub [ 0.811129] usbcore: registered new device driver usb [ 0.812205] pps_core: LinuxPPS API ver. 1 registered [ 0.814015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.817118] PTP clock support registered [ 0.820000] EDAC MC: Ver: 3.0.0 [ 0.822559] PCI: Using ACPI for IRQ routing [ 0.824818] NetLabel: Initializing [ 0.826022] NetLabel: domain hash size = 128 [ 0.827012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.829109] NetLabel: unlabeled traffic allowed by default [ 0.832076] vgaarb: loaded [ 0.833615] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.835018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.845184] clocksource: Switched to clocksource kvm-clock [ 0.964881] VFS: Disk quotas dquot_6.6.0 [ 0.966555] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.968881] *** VALIDATE ramfs *** [ 0.970141] *** VALIDATE hugetlbfs *** [ 0.971502] pnp: PnP ACPI init [ 0.973942] pnp: PnP ACPI: found 6 devices [ 0.989413] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.992306] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.994281] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.996309] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.998452] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.000564] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.003136] NET: Registered protocol family 2 [ 1.005401] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.010450] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.013856] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.018882] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.022656] TCP: Hash tables configured (established 65536 bind 65536) [ 1.025658] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.029095] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.031867] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.034974] NET: Registered protocol family 1 [ 1.040386] RPC: Registered named UNIX socket transport module. [ 1.042461] RPC: Registered udp transport module. [ 1.044200] RPC: Registered tcp transport module. [ 1.045650] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.047578] NET: Registered protocol family 44 [ 1.049272] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.051336] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.053203] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.055383] PCI: CLS 0 bytes, default 64 [ 1.056910] Unpacking initramfs... [ 2.629384] debug: unmapping init [mem 0xffff8fce7cc54000-0xffff8fce7ffbffff] [ 2.634600] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.637584] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.640824] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.149513] Initialise system trusted keyrings [ 3.150816] Key type blacklist registered [ 3.153563] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.168290] zbud: loaded [ 3.171596] *** VALIDATE nfs *** [ 3.172614] *** VALIDATE nfs4 *** [ 3.174291] pstore: using deflate compression [ 3.179565] Platform Keyring initialized [ 3.324262] NET: Registered protocol family 38 [ 3.326650] Key type asymmetric registered [ 3.328497] Asymmetric key parser 'x509' registered [ 3.330444] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.339044] io scheduler mq-deadline registered [ 3.341037] io scheduler kyber registered [ 3.342599] io scheduler bfq registered [ 3.349440] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.352526] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.355180] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.357467] ACPI: Power Button [PWRF] [ 3.363367] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.371293] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.387079] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.396732] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.415837] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.443346] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.472737] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.480616] Non-volatile memory driver v1.3 [ 3.482302] Linux agpgart interface v0.103 [ 3.516292] virtio_blk virtio1: [vda] 145792 512-byte logical blocks (74.6 MB/71.2 MiB) [ 3.519495] vda: detected capacity change from 0 to 74645504 [ 3.556994] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.560854] vdb: detected capacity change from 0 to 1073741824 [ 3.582308] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.586471] vdc: detected capacity change from 0 to 2621440000 [ 3.606243] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.609613] vdd: detected capacity change from 0 to 2621440000 [ 3.628366] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.632577] vde: detected capacity change from 0 to 4294967296 [ 3.654853] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.659304] vdf: detected capacity change from 0 to 4294967296 [ 3.667174] libphy: Fixed MDIO Bus: probed [ 3.674173] usbcore: registered new interface driver usbserial_generic [ 3.677161] usbserial: USB Serial support registered for generic [ 3.680066] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.685679] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.688033] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.691119] mousedev: PS/2 mouse device common for all mice [ 3.694987] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.696479] rtc_cmos 00:05: RTC can wake from S4 [ 3.707240] rtc_cmos 00:05: registered as rtc0 [ 3.710946] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.712388] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.712438] intel_pstate: CPU model not supported [ 3.720246] hid: raw HID events driver (C) Jiri Kosina [ 3.729102] usbcore: registered new interface driver usbhid [ 3.732921] usbhid: USB HID core driver [ 3.735233] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.737059] drop_monitor: Initializing network drop monitor service [ 3.745261] Initializing XFRM netlink socket [ 3.747492] NET: Registered protocol family 10 [ 3.751711] Segment Routing with IPv6 [ 3.753437] NET: Registered protocol family 17 [ 3.755915] mpls_gso: MPLS GSO support [ 3.762426] RAS: Correctable Errors collector initialized. [ 3.764527] AVX version of gcm_enc/dec engaged. [ 3.766293] AES CTR mode by8 optimization enabled [ 3.861898] sched_clock: Marking stable (3861859541, 0)->(5135266396, -1273406855) [ 3.865557] registered taskstats version 1 [ 3.868483] Loading compiled-in X.509 certificates [ 3.871319] zswap: loaded using pool lzo/zbud [ 3.899917] Key type big_key registered [ 3.916376] Key type encrypted registered [ 3.918280] ima: No TPM chip found, activating TPM-bypass! [ 3.920985] ima: Allocated hash algorithm: sha1 [ 3.923202] ima: No architecture policies found [ 3.924863] evm: Initialising EVM extended attributes: [ 3.926903] evm: security.selinux [ 3.928502] evm: security.ima [ 3.929958] evm: security.capability [ 3.931526] evm: HMAC attrs: 0x1 [ 3.934558] rtc_cmos 00:05: setting system clock to 2026-07-14 15:42:32 UTC (1784043752) [ 3.942564] debug: unmapping init [mem 0xffffffffba603000-0xffffffffba7fffff] [ 3.946445] debug: unmapping init [mem 0xffffffffb9382000-0xffffffffb9658fff] [ 3.954327] Write protecting the kernel read-only data: 28672k [ 3.959842] debug: unmapping init [mem 0xffffffffb7a03000-0xffffffffb7bfffff] [ 3.964473] debug: unmapping init [mem 0xffffffffb8314000-0xffffffffb83fffff] [ 4.002818] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.009961] systemd[1]: Detected virtualization kvm. [ 4.012045] systemd[1]: Detected architecture x86-64. [ 4.013566] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.047776] systemd[1]: No hostname configured. [ 4.050659] systemd[1]: Set hostname to . [ 4.055173] random: systemd: uninitialized urandom read (16 bytes read) [ 4.060691] systemd[1]: Initializing machine ID from random generator. [ 4.247428] random: ln: uninitialized urandom read (6 bytes read) [ 4.507912] random: systemd: uninitialized urandom read (16 bytes read) [ 4.512239] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.524687] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 4.535787] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Swap. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK [ 5.220412] random: fast init done ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.390812] device-mapper: uevent: version 1.0.3 [ 6.393984] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ 7.760210] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 8.951945] scsi host0: ata_piix [ 9.008494] scsi host1: ata_piix [ 9.011545] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.014138] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.599545] random: crng init done [ 13.601693] random: 7 urandom warning(s) missed due to ratelimiting [ 16.150712] dracut-initqueue[577]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 17.506894] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.914364] printk: systemd: 26 output lines suppressed due to ratelimiting [ 21.563640] SELinux: Disabled at runtime. [ 21.635794] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.646718] systemd[1]: Detected virtualization kvm. [ 21.649097] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 23.202611] systemd[1]: initrd-switch-root.service: Succeeded. [ 23.206735] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 23.527905] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 23.534751] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 23.541368] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 23.550386] systemd[1]: Starting Journal Service... Starting Journal Service... [ 23.557996] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. [ 24.245869] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Slices. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 26.174567] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 29.165342] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 29.609980] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 31.226418] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [* ] A start job is running for Configur…only root support (10s / no limit)[ 33.121048] EDAC sbridge: Ver: 1.1.2 [** ] A start job is running for Configur…only root support (10s / no limit) [*** ] A start job is running for Configur…only root support (11s / no limit) [ *** ] A start job is running for Configur…only root support (12s / no limit) [ *** ] A start job is running for Configur…only root support (12s / no limit) [ ***] A start job is running for Configur…only root support (13s / no limit) [ **] A start job is running for Configur…only root support (14s / no limit) [ *] A start job is running for Configur…only root support (14s / no limit) [ **] A start job is running for Configur…only root support (15s / no limit) [ ***] A start job is running for Configur…only root support (16s / no limit) [ *** ] A start job is running for Configur…only root support (17s / no limit)[ 40.056437] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (17s / no limit) [*** ] A start job is running for Configur…only root support (18s / no limit)[ 41.222544] NFS: Registering the id_resolver key type [ 41.224882] Key type id_resolver registered [ 41.226684] Key type id_legacy registered [** ] A start job is running for Configur…only root support (19s / no limit) [* ] A start job is running for Configur…only root support (19s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg254-server login: [ 72.029175] hrtimer: interrupt took 3663767 ns [ 112.237757] libcfs: loading out-of-tree module taints kernel. [ 112.275235] Key type ._llcrypt registered [ 112.277482] Key type .llcrypt registered [ 112.356279] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 128.993250] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 130.816637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 130.828777] alg: No test for adler32 (adler32-zlib) [ 132.291284] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 133.289872] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 135.104178] Key type lgssc registered [ 136.707352] Lustre: Echo OBD driver; http://www.lustre.org/ [ 155.963829] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 206.864872] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 220.118588] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 220.193572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 221.596524] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 221.660323] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 221.800336] Lustre: lustre-MDT0000: new disk, initializing [ 221.893948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 221.916106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 227.094475] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 241.376055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 241.512564] Lustre: 6501:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 241.557248] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 241.560870] Lustre: Skipped 1 previous similar message [ 241.671868] Lustre: lustre-MDT0001: new disk, initializing [ 241.734209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 241.758059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 241.762015] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 246.051666] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 250.916740] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 260.829681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 261.061343] Lustre: lustre-OST0000: new disk, initializing [ 261.064492] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 261.069286] Lustre: 8442:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 261.136390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 264.042745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 264.061114] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 264.198160] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 268.458562] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 283.860068] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 284.057250] Lustre: lustre-OST0001: new disk, initializing [ 284.065994] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 284.076830] Lustre: 9515:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 284.143656] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 289.794061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 289.804101] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 289.885534] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 291.203992] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 302.916415] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 311.510328] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 318.619175] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing check_logdir /tmp/testlogs/ [ 324.230492] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing yml_node [ 330.913885] Lustre: DEBUG MARKER: Client: 2.17.54.162 [ 333.581757] Lustre: DEBUG MARKER: MDS: 2.17.54.162 [ 336.016521] Lustre: DEBUG MARKER: OSS: 2.17.54.162 [ 337.949832] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Jul 14 11:48:03 EDT 2026 [ 358.718109] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 360.660698] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 379.873191] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 379.890538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 379.908145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 381.925229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 381.926883] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 383.375673] Lustre: server umount lustre-MDT0000 complete [ 387.492841] LustreError: 6494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784044136 with bad export cookie 4455920024023353314 [ 387.512907] LustreError: 6494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 387.515365] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 392.162221] LustreError: 6510:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 392.163701] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 392.173814] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 392.173822] Lustre: Skipped 3 previous similar messages [ 392.185095] LustreError: 6510:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 392.208106] Lustre: Skipped 3 previous similar messages [ 393.990527] Lustre: server umount lustre-MDT0001 complete [ 399.382255] Lustre: server umount lustre-OST0000 complete [ 405.240230] Lustre: server umount lustre-OST0001 complete [ 413.644355] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 424.538883] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 464.784542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 465.124302] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 465.211910] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 465.364445] Lustre: lustre-MDT0000: new disk, initializing [ 465.570707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 465.590737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 470.841130] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 485.654496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 485.884647] Lustre: 15070:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 486.054464] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 486.071126] Lustre: Skipped 1 previous similar message [ 486.273814] Lustre: lustre-MDT0001: new disk, initializing [ 486.416815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 486.492632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 486.516721] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 493.159811] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 505.501137] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 512.545466] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 521.423252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 521.685844] Lustre: lustre-OST0000: new disk, initializing [ 521.692451] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 521.696595] Lustre: 17042:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 521.784381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 523.744171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 523.755648] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 523.821216] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 531.020279] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 543.090350] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 548.835504] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 549.123186] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 554.513149] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 554.787046] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 559.595649] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 559.599074] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 559.616731] LustreError: Skipped 1 previous similar message [ 559.618034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 559.618042] Lustre: Skipped 1 previous similar message [ 559.675553] Lustre: Skipped 1 previous similar message [ 564.727253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 564.734657] Lustre: Skipped 1 previous similar message [ 571.360281] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 571.565473] Lustre: server umount lustre-OST0000 complete [ 578.531547] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 578.543120] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 578.561518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 578.565307] Lustre: Skipped 3 previous similar messages [ 590.306657] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 590.955281] Lustre: server umount lustre-MDT0000 complete [ 595.423654] LustreError: 15064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784044343 with bad export cookie 4455920024023354840 [ 595.425566] LustreError: 15912:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 595.426029] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 595.458868] LustreError: 15064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 596.114291] Lustre: server umount lustre-MDT0001 complete [ 602.574979] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 11:52:27 (1784044347) [ 605.732760] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 608.499271] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 11:52:33 (1784044353) [ 618.499423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 619.302838] LustreError: 19455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 619.453264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 624.626936] LustreError: 19456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 625.849469] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 628.775623] LustreError: 19455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 635.559988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 639.677909] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 648.469850] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 655.800649] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 663.533532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 670.143399] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 678.623546] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 691.683504] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 691.691077] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 691.695818] Lustre: Skipped 1 previous similar message [ 691.699379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 691.699386] Lustre: Skipped 6 previous similar messages [ 696.354400] Lustre: server umount lustre-OST0000 complete [ 715.233085] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 715.653328] Lustre: server umount lustre-MDT0000 complete [ 717.282969] LustreError: 19455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 717.312867] LustreError: 19455:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 720.447042] LustreError: 19437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784044469 with bad export cookie 4455920024023355729 [ 720.450983] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 720.454961] LustreError: 19437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 721.016655] Lustre: server umount lustre-MDT0001 complete [ 733.438859] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 737.135775] Key type lgssc unregistered [ 737.499978] LNet: 22995:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 737.507932] LNetError: 22995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 737.526643] LNet: Removed LNI 192.168.202.154@tcp [ 738.824324] Key type .llcrypt unregistered [ 738.826785] Key type ._llcrypt unregistered [ 772.852610] Key type ._llcrypt registered [ 772.855218] Key type .llcrypt registered [ 772.989830] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 774.826892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 774.917597] alg: No test for adler32 (adler32-zlib) [ 776.016365] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 776.200277] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 777.874301] Key type lgssc registered [ 778.798892] Lustre: Echo OBD driver; http://www.lustre.org/ [ 787.929734] loop: module loaded [ 788.028181] loop0: detected capacity change from 0 to 204800000 [ 788.283793] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 788.301487] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 788.658519] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 795.636801] loop0: detected capacity change from 0 to 204800000 [ 795.839622] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 795.849829] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 795.859773] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 801.587503] loop0: detected capacity change from 0 to 204800000 [ 808.354323] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 808.413047] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 809.834891] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 809.898711] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 809.960716] Lustre: lustre-MDT0000: new disk, initializing [ 810.052103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 810.071653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 816.512426] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 826.187396] loop1: detected capacity change from 0 to 204800000 [ 833.087095] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 833.453422] Lustre: lustre-OST07c6: new disk, initializing [ 833.465518] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 833.469556] Lustre: Skipped 1 previous similar message [ 833.471949] Lustre: 26189:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 833.582795] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 840.848474] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 841.319651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 841.325416] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 841.382365] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 846.228819] Lustre: 25176:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 846.252625] Lustre: 25176:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 846.268876] Lustre: 25176:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 846.273771] Lustre: 25176:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 846.282731] Lustre: 25176:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 846.297292] Lustre: 25176:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 864.890947] Lustre: 25174:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 864.909798] Lustre: 25174:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 864.917613] Lustre: 25174:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 864.927965] Lustre: 25174:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 864.941765] Lustre: 25174:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 864.949531] Lustre: 25174:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 870.557433] Lustre: 25175:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 870.575069] Lustre: 25175:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 870.582115] Lustre: 25175:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 870.590740] Lustre: 25175:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 870.598740] Lustre: 25175:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 870.613101] Lustre: 25175:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 870.620727] Lustre: 25175:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 870.633746] Lustre: 25175:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 870.647430] Lustre: 25175:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 870.655579] Lustre: 25175:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 870.663960] Lustre: 25175:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 870.674767] Lustre: 25175:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 873.953157] Lustre: 26705:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 873.966144] Lustre: 26705:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 873.974658] Lustre: 26705:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 873.985032] Lustre: 26705:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 874.010338] Lustre: 26705:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 874.022026] Lustre: 26705:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 874.032244] Lustre: 26705:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 874.043419] Lustre: 26705:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 874.051384] Lustre: 26705:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 874.055947] Lustre: 26705:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 874.060814] Lustre: 26705:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 874.065602] Lustre: 26705:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 879.867862] Lustre: 25175:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 879.878224] Lustre: 25175:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 879.899353] Lustre: 25175:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 879.911841] Lustre: 25175:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 879.918208] Lustre: 25175:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 879.930661] Lustre: 25175:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 879.938248] Lustre: 25175:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 879.944209] Lustre: 25175:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 879.950144] Lustre: 25175:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 879.955251] Lustre: 25175:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 879.961142] Lustre: 25175:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 879.968696] Lustre: 25175:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 888.549920] Lustre: 25175:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 888.556053] Lustre: 25175:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 888.560155] Lustre: 25175:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 888.564047] Lustre: 25175:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 888.573082] Lustre: 25175:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 888.577062] Lustre: 25175:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 888.580191] Lustre: 25175:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 888.585064] Lustre: 25175:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 888.589158] Lustre: 25175:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 888.593208] Lustre: 25175:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 888.597813] Lustre: 25175:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 888.602956] Lustre: 25175:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 943.585614] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 943.635936] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 945.964888] Lustre: server umount lustre-OST07c6 complete [ 958.339961] Lustre: server umount lustre-MDT0000 complete [ 976.864263] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 980.876927] Key type lgssc unregistered [ 981.397811] LNet: 28751:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 981.426652] LNetError: 28751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 981.444736] LNet: Removed LNI 192.168.202.154@tcp [ 982.622352] Key type .llcrypt unregistered [ 982.628792] Key type ._llcrypt unregistered [ 1009.380438] Key type ._llcrypt registered [ 1009.381934] Key type .llcrypt registered [ 1009.535398] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 1011.042639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1011.246426] alg: No test for adler32 (adler32-zlib) [ 1012.585892] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 1013.110591] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 1014.832183] Key type lgssc registered [ 1016.384697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1025.562297] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1025.569458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1026.924073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1031.531960] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1037.299798] LustreError: 30024:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1037.339071] LustreError: 30024:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1042.145487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1042.420882] LustreError: 30023:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1042.870117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1048.372192] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1056.517726] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1063.431380] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1071.799608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1072.526940] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1079.728348] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1087.591693] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1097.145709] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1114.597050] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1114.606838] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1114.627038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1117.667212] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1117.677514] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1117.690550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1117.701475] Lustre: Skipped 1 previous similar message [ 1119.065663] Lustre: server umount lustre-OST0000 complete [ 1124.833313] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1124.852473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1124.862225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1127.914246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1127.925281] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1129.371584] Lustre: server umount lustre-MDT0000 complete [ 1133.025613] LustreError: 30023:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1133.038383] LustreError: 30023:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1133.729238] LustreError: 30005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784044882 with bad export cookie 7123837279835028637 [ 1133.734115] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1133.745106] LustreError: 30005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1134.260538] Lustre: server umount lustre-MDT0001 complete [ 1147.748834] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 1150.942389] Key type lgssc unregistered [ 1151.239962] LNet: 33798:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1151.250948] LNetError: 33798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1151.274298] LNet: Removed LNI 192.168.202.154@tcp [ 1152.500967] Key type .llcrypt unregistered [ 1152.502942] Key type ._llcrypt unregistered [ 1173.817367] Key type ._llcrypt registered [ 1173.825598] Key type .llcrypt registered [ 1173.963435] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 1175.567862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1175.597038] alg: No test for adler32 (adler32-zlib) [ 1176.638987] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 1176.854988] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 1178.520213] Key type lgssc registered [ 1180.038113] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1187.330582] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1187.347633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1188.745181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1193.166910] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1199.093783] LustreError: 35066:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1199.128041] LustreError: 35066:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1202.788762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1203.372629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1209.077699] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1218.247359] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1227.070090] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1235.299923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1235.957261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1242.712711] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1252.160384] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1258.576075] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1273.825863] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1273.828130] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1273.861485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1273.868300] Lustre: Skipped 1 previous similar message [ 1278.931442] Lustre: server umount lustre-OST0000 complete [ 1284.092377] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1284.097417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1284.117194] Lustre: Skipped 2 previous similar messages [ 1284.125984] Lustre: Skipped 1 previous similar message [ 1289.188799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1289.196663] Lustre: Skipped 1 previous similar message [ 1294.305438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1294.318210] Lustre: Skipped 1 previous similar message [ 1297.378647] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1297.720192] Lustre: server umount lustre-MDT0000 complete [ 1299.431685] LustreError: 35067:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1299.446485] LustreError: 35067:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1301.851337] LustreError: 35046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784045050 with bad export cookie 13120065213763605925 [ 1301.852890] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1301.860812] LustreError: 35046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1302.323364] Lustre: server umount lustre-MDT0001 complete [ 1313.953191] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 1316.386808] Key type lgssc unregistered [ 1316.643876] LNet: 38625:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1316.650461] LNetError: 38625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1316.665892] LNet: Removed LNI 192.168.202.154@tcp [ 1317.580458] Key type .llcrypt unregistered [ 1317.583683] Key type ._llcrypt unregistered [ 1354.172209] Key type ._llcrypt registered [ 1354.176133] Key type .llcrypt registered [ 1354.396681] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 1356.174085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1356.224413] alg: No test for adler32 (adler32-zlib) [ 1357.208819] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 1357.534239] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 1359.331280] Key type lgssc registered [ 1360.876912] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1371.089862] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1371.151825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1372.686456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1378.375581] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1382.883146] LustreError: 40046:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1382.921752] LustreError: 40046:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1388.006119] LustreError: 40047:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1388.228813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1388.740727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1393.631718] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1404.493724] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1412.232605] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1420.680433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1421.285029] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1427.848630] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1436.334439] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1442.797238] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1455.079164] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1455.099812] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1455.112710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1457.873447] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.54@tcp (stopping) [ 1458.156881] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1458.183469] Lustre: Skipped 1 previous similar message [ 1459.718464] Lustre: server umount lustre-MDT0000 complete [ 1462.951570] LustreError: 42549:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1463.658276] LustreError: 40027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784045212 with bad export cookie 8963225923765191357 [ 1463.660699] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1463.665203] LustreError: 40027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1468.065224] LustreError: 40043:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1468.093800] LustreError: 40043:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1468.134599] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.54@tcp (stopping) [ 1468.143498] Lustre: Skipped 3 previous similar messages [ 1468.402489] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1470.155921] Lustre: server umount lustre-MDT0001 complete [ 1495.153034] Lustre: server umount lustre-OST0000 complete [ 1514.560622] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 1518.030332] Key type lgssc unregistered [ 1518.525873] LNet: 43946:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1518.537711] LNetError: 43946:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1518.565228] LNet: Removed LNI 192.168.202.154@tcp [ 1519.811843] Key type .llcrypt unregistered [ 1519.814291] Key type ._llcrypt unregistered [ 1552.376935] Key type ._llcrypt registered [ 1552.381287] Key type .llcrypt registered [ 1552.496756] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 1553.570164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1553.712875] alg: No test for adler32 (adler32-zlib) [ 1556.039996] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 1556.315551] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 1557.985714] Key type lgssc registered [ 1559.014942] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1566.416703] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1566.442537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1568.012312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1572.852838] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1577.443670] LustreError: 45365:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1577.474970] LustreError: 45365:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1582.575329] LustreError: 45364:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1582.695875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1582.983708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1588.692552] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1598.597498] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1606.445282] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1614.097866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1614.588766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1620.122767] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1627.552273] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1638.149490] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1644.753240] Lustre: lustre-OST0000: Not available for connect from 192.168.202.54@tcp (stopping) [ 1648.613060] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1648.615751] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1648.617119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1648.640204] LustreError: Skipped 1 previous similar message [ 1648.681113] Lustre: Skipped 1 previous similar message [ 1649.826630] Lustre: lustre-OST0000: Not available for connect from 192.168.202.54@tcp (stopping) [ 1649.838559] Lustre: Skipped 1 previous similar message [ 1650.622031] Lustre: server umount lustre-OST0000 complete [ 1669.097347] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1669.120522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1669.129066] Lustre: Skipped 1 previous similar message [ 1675.246225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1675.259248] Lustre: Skipped 3 previous similar messages [ 1675.381112] Lustre: server umount lustre-MDT0000 complete [ 1679.102156] LustreError: 46099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784045427 with bad export cookie 6284589147529186447 [ 1679.108465] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1679.121574] LustreError: 46099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1685.685763] Lustre: server umount lustre-MDT0001 complete [ 1697.846746] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 1700.442586] Key type lgssc unregistered [ 1700.798674] LNet: 49097:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1700.841772] LNetError: 49097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1701.862600] LNet: Removed LNI 192.168.202.154@tcp [ 1702.752274] Key type .llcrypt unregistered [ 1702.754267] Key type ._llcrypt unregistered [ 1725.854514] Key type ._llcrypt registered [ 1725.857740] Key type .llcrypt registered [ 1725.948921] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 1726.917489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1727.018124] alg: No test for adler32 (adler32-zlib) [ 1728.316107] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 1728.535725] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 1730.176579] Key type lgssc registered [ 1731.428753] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1738.265515] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1738.288304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1739.709726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1743.856918] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1749.993434] LustreError: 50512:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1750.024257] LustreError: 50512:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1754.177589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1754.497611] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1761.020388] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1770.989338] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1778.333763] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1785.978974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1786.764432] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1794.716142] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1803.045376] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1810.321692] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1820.559233] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1824.430786] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1837.146681] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-07-1412h13m02s [ 1838.779951] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1840.843923] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.54@tcp (stopping) [ 1841.130092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1841.152671] Lustre: Skipped 1 previous similar message [ 1841.893297] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.54@tcp (stopping) [ 1841.902470] Lustre: Skipped 3 previous similar messages [ 1846.946372] LustreError: 53004:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1847.191248] Lustre: server umount lustre-MDT0000 complete [ 1848.296612] LustreError: 53004:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1848.312928] LustreError: 53004:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1851.060566] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1853.095032] LustreError: 50509:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1859.242969] LustreError: 53004:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1859.279396] LustreError: 53004:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1859.494736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1859.697879] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1860.066268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1860.142210] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1861.112978] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1866.019748] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1870.268460] Lustre: DEBUG MARKER: Wait for df (36732) ... [ 1872.129985] Lustre: DEBUG MARKER: done [ 1876.456602] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1876.479470] Lustre: Skipped 1 previous similar message [ 1876.486277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1876.490317] Lustre: Skipped 3 previous similar messages [ 1881.571969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1881.584702] Lustre: Skipped 2 previous similar messages [ 1882.689826] Lustre: server umount lustre-OST0000 complete [ 1891.824153] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1891.825393] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1891.842610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1891.873441] Lustre: Skipped 2 previous similar messages [ 1893.764134] Lustre: server umount lustre-MDT0000 complete [ 1898.235384] LustreError: 50492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784045646 with bad export cookie 13363765134541133045 [ 1898.249255] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1898.251120] LustreError: 50492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1898.536968] Lustre: server umount lustre-MDT0001 complete [ 1911.606257] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 1915.187169] Key type lgssc unregistered [ 1915.601898] LNet: 55459:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1915.632245] LNetError: 55459:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1915.649674] LNet: Removed LNI 192.168.202.154@tcp [ 1916.620870] Key type .llcrypt unregistered [ 1916.623987] Key type ._llcrypt unregistered [ 1961.387631] Key type ._llcrypt registered [ 1961.389318] Key type .llcrypt registered [ 1961.506150] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 1963.066812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1963.122170] alg: No test for adler32 (adler32-zlib) [ 1964.389768] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 1964.720455] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 1966.384197] Key type lgssc registered [ 1967.331574] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1975.057671] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1975.100829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1976.371820] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1976.396033] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1976.557025] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1981.186834] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1990.861677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1991.012912] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1991.070902] Lustre: 57670:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1991.318346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1996.199304] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2007.863339] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2015.228303] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2022.834385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2022.936098] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2023.070402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2031.288696] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2032.830332] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 2041.260056] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2049.673849] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-07-1412h16m35s [ 2051.865068] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 2061.570812] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 2065.424804] Lustre: DEBUG MARKER: Creating a test file and stat it [ 2065.453774] LustreError: 57677:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 2085.560735] Lustre: lustre-MDT0000: Client 1f47ce0f-ccf1-40b1-9846-617cd076af25 (at 192.168.202.54@tcp) reconnecting [ 2105.520181] LustreError: 57677:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 2107.842380] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2110.733281] Lustre: DEBUG MARKER: done [ 2120.163252] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2120.170638] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2120.179964] LustreError: Skipped 1 previous similar message [ 2120.184830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2120.191236] Lustre: Skipped 1 previous similar message [ 2120.231215] Lustre: Skipped 1 previous similar message [ 2122.872888] Lustre: server umount lustre-OST0000 complete [ 2129.892485] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2129.909346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2129.924947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2130.983695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2130.996400] Lustre: Skipped 2 previous similar messages [ 2137.077500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2137.083920] Lustre: Skipped 1 previous similar message [ 2141.158461] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2141.218432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2141.228301] Lustre: Skipped 1 previous similar message [ 2141.391699] Lustre: server umount lustre-MDT0000 complete [ 2145.762853] LustreError: 58427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784045894 with bad export cookie 4747235468293700561 [ 2145.769294] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2145.775983] LustreError: 58427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2146.175666] Lustre: server umount lustre-MDT0001 complete [ 2160.795373] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 2164.311926] Key type lgssc unregistered [ 2164.618694] LNet: 62004:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2164.629382] LNetError: 62004:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2165.682145] LNet: Removed LNI 192.168.202.154@tcp [ 2166.798196] Key type .llcrypt unregistered [ 2166.801182] Key type ._llcrypt unregistered [ 2221.476890] Key type ._llcrypt registered [ 2221.478722] Key type .llcrypt registered [ 2221.586823] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 2223.389543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2223.426394] alg: No test for adler32 (adler32-zlib) [ 2224.483485] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 2224.748109] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 2226.424212] Key type lgssc registered [ 2227.598215] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2231.059724] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2231.101330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2232.345822] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2232.398500] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2232.728591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2233.859086] Lustre: Failing over lustre-MDT0000 [ 2234.510476] Lustre: server umount lustre-MDT0000 complete [ 2242.517465] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 12:19:48 (1784045988) [ 2250.911095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2251.651033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2256.788616] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2266.826907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2266.943125] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2267.000557] Lustre: 64806:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2267.266261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2273.154585] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2283.095652] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2290.002950] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2297.804709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2298.110556] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2298.343865] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2306.229921] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2307.235338] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2315.203483] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2321.001937] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2328.546672] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2328.557970] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2328.570477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2333.153637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2333.169849] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2333.192744] Lustre: Skipped 1 previous similar message [ 2334.492790] Lustre: server umount lustre-MDT0000 complete [ 2338.485566] LustreError: 65560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784046087 with bad export cookie 7848954918434900688 [ 2338.491897] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2338.502100] LustreError: 65560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2339.009871] Lustre: server umount lustre-MDT0001 complete [ 2342.949995] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2347.693768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2356.933776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2358.752263] Lustre: 63728:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784046091/real 1784046091] req@000000004a1aac19 x1870707790227712/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784046107 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2358.789180] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2364.393231] LustreError: 66666:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2364.406288] LustreError: 66666:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2364.519564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2364.671860] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2365.472116] Lustre: 63728:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784046097/real 1784046097] req@00000000973bdcd1 x1870707790227968/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784046113 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2367.524981] LustreError: 66667:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2368.612232] LustreError: 68590:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2368.657146] LustreError: 68590:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2370.529591] Lustre: 63725:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784046102/real 1784046102] req@0000000049a4c211 x1870707790228096/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784046118 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2371.786896] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2374.120959] LustreError: 68753:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2374.134900] LustreError: 68753:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2375.651788] Lustre: 63726:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784046107/real 1784046107] req@000000006542bfef x1870707790228224/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784046123 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2379.238144] LustreError: 66667:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2379.259643] LustreError: 66667:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2383.406848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2383.883053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2387.955472] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2389.307063] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2399.350414] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2406.483356] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2414.562535] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2414.574136] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2414.586519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2414.588459] Lustre: Skipped 2 previous similar messages [ 2418.657902] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2418.662358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2418.682415] Lustre: Skipped 1 previous similar message [ 2418.699460] Lustre: Skipped 2 previous similar messages [ 2420.736504] Lustre: server umount lustre-MDT0000 complete [ 2423.780212] LustreError: 68753:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2423.802540] LustreError: 68753:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2424.891515] LustreError: 64797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784046173 with bad export cookie 7848954918434902970 [ 2424.899325] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2424.903921] LustreError: 64797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2425.190654] Lustre: server umount lustre-MDT0001 complete [ 2432.387075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2442.906704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2445.281103] Lustre: 63727:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784046177/real 1784046177] req@000000001936a0af x1870707790257152/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784046193 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2445.317210] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2449.896090] LustreError: 66667:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2450.014138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2450.105344] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2454.685454] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2456.098801] Lustre: 63728:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784046188/real 1784046188] req@00000000c2804fe4 x1870707790257792/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784046204 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2456.121025] Lustre: 63728:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 2464.430984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2469.698221] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2469.868962] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2469.872716] Lustre: Skipped 1 previous similar message [ 2474.997414] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2478.999272] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2485.634885] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2495.463248] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2495.464105] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2495.473757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2505.700463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2505.714548] Lustre: Skipped 7 previous similar messages [ 2507.233162] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2507.491340] Lustre: server umount lustre-MDT0000 complete [ 2510.827281] LustreError: 72266:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2510.865899] LustreError: 72266:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 2511.790806] LustreError: 64799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784046260 with bad export cookie 7848954918434904349 [ 2511.795317] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2511.803206] LustreError: 64799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2512.166348] Lustre: server umount lustre-MDT0001 complete [ 2519.036536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2522.933527] Lustre: DEBUG MARKER: files compared the same [ 2534.933333] Lustre: server umount lustre-OST0000 complete [ 2556.349825] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 2558.225567] Key type lgssc unregistered [ 2558.441253] LNet: 74550:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2558.455374] LNetError: 74550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2558.470699] LNet: Removed LNI 192.168.202.154@tcp [ 2559.330725] Key type .llcrypt unregistered [ 2559.333376] Key type ._llcrypt unregistered [ 2589.423808] Key type ._llcrypt registered [ 2589.426213] Key type .llcrypt registered [ 2589.525395] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 2590.475476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2590.497510] alg: No test for adler32 (adler32-zlib) [ 2591.481747] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 2591.652521] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 2593.320190] Key type lgssc registered [ 2594.113765] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2600.924111] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2600.944932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2602.477498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2606.949547] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug +malloc all [ 2612.711402] LustreError: 75981:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2612.733991] LustreError: 75981:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2615.539276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2615.934426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2621.182481] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug +malloc all [ 2629.684511] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2636.899283] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2646.014686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2646.333919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2648.395329] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2654.048900] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug +malloc all [ 2664.254663] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2676.202199] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2676.204258] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2676.208286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2676.209885] Lustre: Skipped 1 previous similar message [ 2681.314770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2681.321293] Lustre: Skipped 1 previous similar message [ 2681.760205] Lustre: server umount lustre-OST0000 complete [ 2687.458532] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2687.470056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2687.497487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2687.507877] Lustre: Skipped 1 previous similar message [ 2691.555209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2691.555882] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2693.407876] Lustre: server umount lustre-MDT0000 complete [ 2696.679809] LustreError: 75975:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2696.705207] LustreError: 75975:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2698.126248] LustreError: 78762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784046446 with bad export cookie 12966923591237546381 [ 2698.126437] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2698.143857] LustreError: 78762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2698.721253] Lustre: server umount lustre-MDT0001 complete [ 2710.866780] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 2713.678831] Key type lgssc unregistered [ 2714.007727] LNet: 79516:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2714.021478] LNetError: 79516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2714.070878] LNet: Removed LNI 192.168.202.154@tcp [ 2714.981440] Key type .llcrypt unregistered [ 2714.984083] Key type ._llcrypt unregistered [ 2741.345459] Key type ._llcrypt registered [ 2741.348789] Key type .llcrypt registered [ 2741.464588] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 2742.551666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2742.765808] alg: No test for adler32 (adler32-zlib) [ 2743.843440] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 2744.136376] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 2745.856966] Key type lgssc registered [ 2747.184650] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2756.728360] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2756.745468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2772.385441] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2777.568327] LustreError: 80908:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.154@tcp: failed processing log, type 4: rc = -110 [ 2804.320396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2812.523886] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2822.393468] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2831.982448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2832.155146] Lustre: *** cfs_fail_loc=706, val=0*** [ 2832.761237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2832.882083] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2837.991755] LustreError: 80933:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2837.999793] LustreError: 80933:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2838.005898] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2839.020166] LustreError: 81993:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2839.042131] LustreError: 81993:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2844.140320] LustreError: 81994:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2844.171211] LustreError: 81994:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2847.106755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2847.464199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2851.842331] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2860.782617] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2866.880292] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2873.320551] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2873.333581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2873.334395] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2873.345719] Lustre: Skipped 1 previous similar message [ 2873.368860] Lustre: Skipped 1 previous similar message [ 2875.067431] Lustre: server umount lustre-OST0000 complete [ 2883.554315] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2883.569350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2883.577210] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2883.584327] Lustre: Skipped 1 previous similar message [ 2888.673776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2888.679131] Lustre: Skipped 1 previous similar message [ 2893.280388] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2893.634893] Lustre: server umount lustre-MDT0000 complete [ 2893.799962] LustreError: 83774:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2893.843723] LustreError: 83774:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2898.273251] LustreError: 80914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784046646 with bad export cookie 8915468519373729233 [ 2898.285376] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2898.301199] LustreError: 80914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2898.929978] Lustre: server umount lustre-MDT0001 complete [ 2911.602761] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 2914.646211] Key type lgssc unregistered [ 2914.938454] LNet: 84528:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2914.943957] LNetError: 84528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2914.977607] LNet: Removed LNI 192.168.202.154@tcp [ 2915.925495] Key type .llcrypt unregistered [ 2915.930481] Key type ._llcrypt unregistered [ 2942.644071] Key type ._llcrypt registered [ 2942.646187] Key type .llcrypt registered [ 2942.781849] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 2944.754541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2944.834095] alg: No test for adler32 (adler32-zlib) [ 2946.003447] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 2946.323924] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 2948.128619] Key type lgssc registered [ 2949.274161] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2956.696533] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2956.729595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2963.186742] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2971.578068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2972.273589] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2977.764652] LustreError: 86635:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2977.781988] LustreError: 86635:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2977.850032] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2981.859686] LustreError: 86639:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2981.874101] LustreError: 86639:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2986.981352] LustreError: 86635:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2987.016581] LustreError: 86635:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2987.640703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2988.082727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2993.134788] LustreError: 86639:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2993.176960] LustreError: 86639:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2995.538976] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2997.230576] LustreError: 86638:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2997.260749] LustreError: 86638:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3006.640369] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3006.661422] Lustre: MGS: Client 3355d20a-939f-4b99-92d3-c70a6ca7df25 (at 0@lo) reconnecting [ 3006.694551] LustreError: 86649:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3007.069614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3007.136642] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 3011.722567] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3028.962985] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3028.974335] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3028.999840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3032.550833] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3032.550836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3032.550865] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3033.957152] Lustre: server umount lustre-OST0000 complete [ 3042.792535] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3042.808995] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3042.814048] Lustre: Skipped 1 previous similar message [ 3042.815833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3042.840311] Lustre: Skipped 2 previous similar messages [ 3044.332090] Lustre: server umount lustre-MDT0000 complete [ 3047.904936] LustreError: 86638:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3047.928242] LustreError: 86638:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3048.777792] Lustre: server umount lustre-MDT0001 complete [ 3052.544579] Lustre: server umount MGS complete [ 3064.803391] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 12:33:30 (1784046810) [ 3096.252810] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 3107.883951] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 3148.641596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3153.608724] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3164.557590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3164.826660] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3164.884159] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3164.985699] Lustre: lustre-MDT0001: new disk, initializing [ 3165.078592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3165.115111] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3165.126498] Lustre: 93830:0:(lod_sub_object.c:971:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3165.131368] LustreError: 93830:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3170.324206] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3182.238403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3182.530773] Lustre: lustre-OST0000: new disk, initializing [ 3182.538661] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3182.548771] Lustre: 94594:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3182.617123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3188.837340] Lustre: 94864:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000037d55285 x1870708546251520/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1784046953 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3189.434284] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3200.043818] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3207.748969] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3207.762210] Lustre: MGS: Client 740a0dd3-a69f-47f3-bd7b-1382158b761a (at 0@lo) reconnecting [ 3207.783838] LustreError: 93816:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3207.833961] Lustre: 93099:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3207.904989] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3207.977315] Lustre: lustre-MDT0000: new disk, initializing [ 3208.203742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3208.235679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3208.252261] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3209.313324] Lustre: 95664:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000008355682e x1870708546258560/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1784046973 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3213.165070] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3214.406831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3214.417886] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3214.472961] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3239.907963] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3239.913047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3239.915668] Lustre: Skipped 1 previous similar message [ 3239.920548] Lustre: Skipped 1 previous similar message [ 3244.949731] Lustre: server umount lustre-OST0000 complete [ 3250.147424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3250.150048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3250.161181] Lustre: Skipped 1 previous similar message [ 3250.171803] Lustre: Skipped 1 previous similar message [ 3254.578911] Lustre: server umount lustre-MDT0000 complete [ 3255.272096] LustreError: 95914:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3255.295532] LustreError: 95914:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3273.184131] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3273.492348] Lustre: server umount lustre-MDT0001 complete [ 3277.365746] Lustre: server umount MGS complete [ 3289.630838] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 12:37:15 (1784047035) [ 3307.524676] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 3310.394939] Key type lgssc unregistered [ 3310.796779] LNet: 98167:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3310.808389] LNetError: 98167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3310.829222] LNet: Removed LNI 192.168.202.154@tcp [ 3311.787065] Key type .llcrypt unregistered [ 3311.788244] Key type ._llcrypt unregistered [ 3327.837443] Key type ._llcrypt registered [ 3327.841557] Key type .llcrypt registered [ 3327.951922] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 3329.041645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3329.078068] alg: No test for adler32 (adler32-zlib) [ 3330.118226] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 3330.323486] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 3332.040470] Key type lgssc registered [ 3333.143140] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3342.047152] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3342.060850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3343.633404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3348.035988] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3354.087678] LustreError: 99390:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3354.130778] LustreError: 99390:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3356.544508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3356.894491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3361.308758] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3368.536207] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3373.665615] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3380.909282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3381.221653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3388.132228] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3390.494546] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3395.867238] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3405.796917] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3405.797695] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3405.804746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3405.824234] Lustre: Skipped 1 previous similar message [ 3410.916510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3410.927685] Lustre: Skipped 1 previous similar message [ 3411.832554] Lustre: server umount lustre-OST0000 complete [ 3418.594684] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3418.600463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3418.612054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3418.618359] Lustre: Skipped 1 previous similar message [ 3421.157217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3421.166854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3422.780347] Lustre: server umount lustre-MDT0000 complete [ 3426.283692] LustreError: 99390:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3426.315654] LustreError: 99390:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3427.066034] LustreError: 99372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784047175 with bad export cookie 14929487631284619260 [ 3427.073142] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3427.076396] LustreError: 99372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3427.383131] Lustre: server umount lustre-MDT0001 complete [ 3440.999353] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 3444.403898] Key type lgssc unregistered [ 3444.834711] LNet: 102925:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3444.847350] LNetError: 102925:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3444.881921] LNet: Removed LNI 192.168.202.154@tcp [ 3445.881553] Key type .llcrypt unregistered [ 3445.884208] Key type ._llcrypt unregistered [ 3461.045970] Key type ._llcrypt registered [ 3461.048646] Key type .llcrypt registered [ 3461.164441] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 3462.464683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3462.504777] alg: No test for adler32 (adler32-zlib) [ 3463.478564] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 3463.692699] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 3465.480208] Key type lgssc registered [ 3466.651067] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3472.083912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3472.090540] LustreError: 104011:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3472.093337] Lustre: Skipped 1 previous similar message [ 3472.104857] LustreError: 104010:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3472.108900] LustreError: 104011:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4994 [ 3472.114186] LustreError: 104011:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3472.128802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3472.129576] LustreError: 104011:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3473.510522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3479.395645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3479.663742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3483.817727] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3489.349351] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing lsmod [ 3492.287567] LustreError: 105199:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3492.396372] LustreError: 105215:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3492.409351] LustreError: 105199:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4880 [ 3492.427880] LustreError: 105199:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3492.443410] LustreError: 105199:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3492.458938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3492.828536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3496.429734] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3496.441208] Lustre: Skipped 2 previous similar messages [ 3496.443974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3496.449434] Lustre: Skipped 1 previous similar message [ 3501.540540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3501.549788] Lustre: Skipped 3 previous similar messages [ 3503.045672] Lustre: server umount lustre-MDT0000 complete [ 3504.614943] LustreError: 104047:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3504.639512] LustreError: 104047:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3509.050817] LustreError: 105401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784047257 with bad export cookie 5024189689139704910 [ 3509.056115] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3509.071705] LustreError: 105401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3509.106252] Lustre: 104041:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.154@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3509.729508] LustreError: 104043:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3509.730339] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3509.779178] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3510.886646] LustreError: 104478:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000b622ee74 x1870709089389184/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3511.031950] Lustre: server umount lustre-MDT0001 complete [ 3515.343909] Lustre: server umount lustre-OST0000 complete [ 3524.094746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3524.533390] LustreError: 106278:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3524.600214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3529.699328] LustreError: 106279:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3531.020838] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3534.821684] LustreError: 106278:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3541.374515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3541.871457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3547.282291] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3554.656382] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3559.983335] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3566.548234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3566.940658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3570.045616] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3574.120912] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3582.325376] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3597.795913] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3597.800941] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3597.811413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3602.735582] Lustre: server umount lustre-OST0000 complete [ 3608.038070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3608.044270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3608.065637] Lustre: Skipped 2 previous similar messages [ 3608.077647] Lustre: Skipped 2 previous similar messages [ 3618.276340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3618.279674] Lustre: Skipped 2 previous similar messages [ 3621.344177] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3621.553353] Lustre: server umount lustre-MDT0000 complete [ 3623.401886] LustreError: 106274:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3623.431089] LustreError: 106274:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3625.591388] LustreError: 106260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784047374 with bad export cookie 5024189689139705589 [ 3625.612723] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3625.613968] LustreError: 106260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3625.929053] Lustre: server umount lustre-MDT0001 complete [ 3639.804379] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 3643.196386] Key type lgssc unregistered [ 3643.483090] LNet: 109839:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3643.486914] LNetError: 109839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3643.499103] LNet: Removed LNI 192.168.202.154@tcp [ 3644.639361] Key type .llcrypt unregistered [ 3644.641042] Key type ._llcrypt unregistered [ 3675.769751] Key type ._llcrypt registered [ 3675.772928] Key type .llcrypt registered [ 3675.887093] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 3676.890648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3676.919570] alg: No test for adler32 (adler32-zlib) [ 3677.906431] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 3678.169853] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 3679.840184] Key type lgssc registered [ 3680.970724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3689.109779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3689.138864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3690.646515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3694.730373] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3701.220708] LustreError: 111254:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3701.244883] LustreError: 111254:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3704.074795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3704.400731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3709.212665] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3717.761580] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3724.700699] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3732.333968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3732.767996] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3735.885574] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3740.015095] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3749.358947] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3755.650208] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3760.181574] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3764.711336] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3764.717344] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3764.732305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3764.762057] Lustre: Skipped 1 previous similar message [ 3769.836736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3769.847314] Lustre: Skipped 1 previous similar message [ 3770.675056] Lustre: server umount lustre-OST0000 complete [ 3775.972224] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3775.985456] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3775.997950] Lustre: Skipped 1 previous similar message [ 3776.008550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3780.073973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3780.075799] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3781.697894] Lustre: server umount lustre-MDT0000 complete [ 3785.195172] LustreError: 111250:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3785.219882] LustreError: 111250:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3786.125357] LustreError: 111236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784047534 with bad export cookie 462305866287786415 [ 3786.131531] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3786.138862] LustreError: 111236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3786.590815] Lustre: server umount lustre-MDT0001 complete [ 3799.157571] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 3802.013983] Key type lgssc unregistered [ 3802.403850] LNet: 114892:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3802.415342] LNetError: 114892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3802.447710] LNet: Removed LNI 192.168.202.154@tcp [ 3803.567951] Key type .llcrypt unregistered [ 3803.569878] Key type ._llcrypt unregistered [ 3821.497410] Key type ._llcrypt registered [ 3821.498756] Key type .llcrypt registered [ 3821.618195] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 3822.493335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3822.602631] alg: No test for adler32 (adler32-zlib) [ 3823.776965] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 3824.098104] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 3825.808215] Key type lgssc registered [ 3827.093641] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3836.396328] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3836.408638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3837.845186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3843.081405] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3848.171580] LustreError: 116114:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3848.203522] LustreError: 116114:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3852.591085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3853.289124] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3858.598315] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3867.217508] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3875.520906] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3884.615951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3885.226453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3892.582687] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3901.534123] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3907.772782] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3918.305787] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3918.312681] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3918.316411] LustreError: Skipped 1 previous similar message [ 3918.317992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3918.351775] Lustre: Skipped 1 previous similar message [ 3922.766684] Lustre: server umount lustre-OST0000 complete [ 3928.554313] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3928.573416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3928.577855] Lustre: Skipped 1 previous similar message [ 3928.596395] Lustre: Skipped 2 previous similar messages [ 3933.590959] Lustre: server umount lustre-MDT0000 complete [ 3933.684901] LustreError: 116114:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3933.717803] LustreError: 116114:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3937.370229] LustreError: 116095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784047685 with bad export cookie 907262847500023445 [ 3937.392149] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3937.410128] LustreError: 116095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3937.844742] Lustre: server umount lustre-MDT0001 complete [ 3950.710258] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 3953.561928] Key type lgssc unregistered [ 3953.951134] LNet: 119674:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3953.961113] LNetError: 119674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3953.981142] LNet: Removed LNI 192.168.202.154@tcp [ 3955.148778] Key type .llcrypt unregistered [ 3955.151058] Key type ._llcrypt unregistered [ 3983.076149] Key type ._llcrypt registered [ 3983.077989] Key type .llcrypt registered [ 3983.196406] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 3984.626822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3984.672216] alg: No test for adler32 (adler32-zlib) [ 3985.712760] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 3985.959135] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 3987.648311] Key type lgssc registered [ 3988.636878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3996.267914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3996.290219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3997.701526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4001.787294] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4007.917879] LustreError: 121090:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4007.929370] LustreError: 121090:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4011.341319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4011.753067] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4016.900741] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4024.171829] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4030.560865] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4037.711250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4038.209616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4043.305071] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 4044.592961] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4052.699718] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4063.601628] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 4067.164705] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 4067.169502] Lustre: Skipped 2 previous similar messages [ 4070.984740] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 4071.018197] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 4071.253427] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 4071.255928] Lustre: Skipped 2 previous similar messages [ 4080.757766] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4080.760882] Lustre: Skipped 1 previous similar message [ 4085.861432] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.54@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 4085.870847] Lustre: Skipped 2 previous similar messages [ 4086.133152] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.54@tcp 192.168.0.[2,10]@tcp [ 4096.026855] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.54/24@tcp in log lustre-MDT0000 [ 4096.031968] Lustre: Skipped 2 previous similar messages [ 4102.290841] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.54/24@tcp [ 4102.294280] Lustre: Skipped 1 previous similar message [ 4109.797556] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4109.802979] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4109.843403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4112.867823] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4112.872089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4112.884232] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4112.920545] Lustre: Skipped 1 previous similar message [ 4116.000856] Lustre: server umount lustre-OST0000 complete [ 4123.109642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4123.119104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4123.133640] Lustre: Skipped 1 previous similar message [ 4126.846683] Lustre: server umount lustre-MDT0000 complete [ 4128.234209] LustreError: 122088:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4128.255056] LustreError: 122088:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4131.693763] LustreError: 121073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784047880 with bad export cookie 16248506522553375949 [ 4131.694503] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4131.709160] LustreError: 121073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4132.175808] Lustre: server umount lustre-MDT0001 complete [ 4145.305347] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 4147.804695] Key type lgssc unregistered [ 4148.115811] LNet: 126037:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4148.124055] LNetError: 126037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4148.139714] LNet: Removed LNI 192.168.202.154@tcp [ 4149.455647] Key type .llcrypt unregistered [ 4149.460749] Key type ._llcrypt unregistered [ 4185.460339] Key type ._llcrypt registered [ 4185.462413] Key type .llcrypt registered [ 4185.617390] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 4188.454244] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4188.665581] alg: No test for adler32 (adler32-zlib) [ 4190.046842] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 4190.357380] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 4192.096838] Key type lgssc registered [ 4194.311937] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4204.557885] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4204.591218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4206.023769] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4206.042865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4206.045317] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.54/24@tcp [ 4210.025406] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4216.309954] LustreError: 127601:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4216.326815] LustreError: 127601:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4218.933739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4219.178696] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4219.212115] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.54/24@tcp [ 4219.286410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4224.184493] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4232.817887] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4238.904689] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4245.868790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4246.361570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4250.505153] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 4252.823117] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4261.446981] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4274.148865] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4274.151554] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4274.163652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4274.179399] Lustre: Skipped 1 previous similar message [ 4279.209837] Lustre: server umount lustre-OST0000 complete [ 4284.388495] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4284.405946] Lustre: Skipped 1 previous similar message [ 4284.415115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4284.421419] Lustre: Skipped 1 previous similar message [ 4289.225134] Lustre: server umount lustre-MDT0000 complete [ 4289.506074] LustreError: 127620:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4292.852644] LustreError: 128344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784048041 with bad export cookie 10654885691619936547 [ 4292.855467] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4292.862415] LustreError: 128344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4293.105573] Lustre: server umount lustre-MDT0001 complete [ 4303.799344] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 4307.264719] Key type lgssc unregistered [ 4307.616792] LNet: 131193:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4307.620743] LNetError: 131193:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4307.642768] LNet: Removed LNI 192.168.202.154@tcp [ 4308.679582] Key type .llcrypt unregistered [ 4308.682538] Key type ._llcrypt unregistered [ 4336.601242] Key type ._llcrypt registered [ 4336.605985] Key type .llcrypt registered [ 4336.712141] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 4353.907973] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 4355.354470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4355.402689] alg: No test for adler32 (adler32-zlib) [ 4356.411319] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 4356.671255] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 4358.360283] Key type lgssc registered [ 4359.599860] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4398.293374] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4398.330861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4399.705319] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4399.746115] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4399.899371] Lustre: lustre-MDT0000: new disk, initializing [ 4400.057517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4400.077473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4404.914240] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4416.331503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4416.524452] Lustre: 134595:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4416.608231] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4416.614452] Lustre: Skipped 1 previous similar message [ 4416.731723] Lustre: lustre-MDT0001: new disk, initializing [ 4416.894336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4416.964129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4416.975815] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4423.179794] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4434.287770] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4439.704798] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4447.073321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4447.287687] Lustre: lustre-OST0000: new disk, initializing [ 4447.290853] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4447.298995] Lustre: 136563:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4447.383766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4454.008634] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4455.037975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4455.048505] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4455.171300] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4464.103905] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4469.120400] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4469.314167] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4474.540708] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4474.781931] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4481.004921] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4481.025920] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4481.029395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4481.038846] Lustre: Skipped 1 previous similar message [ 4486.136898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4486.143101] Lustre: Skipped 1 previous similar message [ 4491.233347] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4491.261775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4491.275571] Lustre: Skipped 1 previous similar message [ 4491.979151] Lustre: server umount lustre-OST0000 complete [ 4498.915663] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4498.932629] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4498.989845] Lustre: Skipped 1 previous similar message [ 4499.001315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4501.475757] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4505.070252] Lustre: server umount lustre-MDT0000 complete [ 4509.917864] LustreError: 134585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784048258 with bad export cookie 16834690160104167115 [ 4509.921219] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4509.929350] LustreError: 134585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4510.407717] Lustre: server umount lustre-MDT0001 complete [ 4520.056665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4520.373252] LustreError: 138783:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4520.389083] LustreError: 138783:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4520.428205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4524.260649] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4525.544050] LustreError: 138783:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4529.638542] LustreError: 138784:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4532.366879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4532.813031] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4537.641667] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4543.997662] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4548.916249] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4555.340989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4555.556623] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4560.917336] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4569.209383] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4574.342960] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4574.572201] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4579.303979] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4579.463499] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4587.889753] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4588.142747] Lustre: lustre-OST0001: new disk, initializing [ 4588.148641] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4588.161554] Lustre: 142156:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4588.311064] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4594.337783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4594.359326] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4594.524212] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4594.977939] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4602.565697] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4602.790510] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4608.219789] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4608.514756] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4615.406798] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8bc7c81b5000.ost_server_uuid 50 [ 4616.897524] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8bc7c81b5000.ost_server_uuid in IDLE state after 0 sec [ 4630.517818] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4630.523982] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4630.531246] LustreError: Skipped 1 previous similar message [ 4630.533609] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4630.544927] Lustre: Skipped 1 previous similar message [ 4630.552851] Lustre: Skipped 3 previous similar messages [ 4633.687545] Lustre: server umount lustre-OST0001 complete [ 4635.626863] LustreError: 140607:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4635.651890] LustreError: 140607:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4639.724553] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4639.745903] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4639.769728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4640.752992] LustreError: 140607:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4640.783114] LustreError: 140607:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4650.980364] LustreError: 141124:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4651.004981] LustreError: 141124:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4653.024184] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4653.179671] Lustre: server umount lustre-OST0000 complete [ 4660.717137] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4660.720062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4660.753094] Lustre: Skipped 1 previous similar message [ 4660.760704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4660.772216] Lustre: Skipped 6 previous similar messages [ 4663.823689] Lustre: server umount lustre-MDT0000 complete [ 4668.370784] LustreError: 143461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784048416 with bad export cookie 16834690160104167997 [ 4668.381102] LustreError: 143461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4668.382804] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4668.719914] Lustre: server umount lustre-MDT0001 complete [ 4689.412131] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 4692.045595] Key type lgssc unregistered [ 4692.333661] LNet: 145164:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4692.337741] LNetError: 145164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4692.349359] LNet: Removed LNI 192.168.202.154@tcp [ 4693.121072] Key type .llcrypt unregistered [ 4693.123912] Key type ._llcrypt unregistered [ 4736.367471] Key type ._llcrypt registered [ 4736.370070] Key type .llcrypt registered [ 4736.525133] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 4755.793778] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 4757.313065] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4757.374040] alg: No test for adler32 (adler32-zlib) [ 4758.394870] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 4758.589334] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 4760.232450] Key type lgssc registered [ 4761.706184] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4799.139897] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4799.205285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4800.471436] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4800.503325] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4800.579512] Lustre: lustre-MDT0000: new disk, initializing [ 4800.646482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4800.661114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4805.285415] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4816.676049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4816.862746] Lustre: 149333:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4816.945368] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4816.949500] Lustre: Skipped 1 previous similar message [ 4817.054199] Lustre: lustre-MDT0001: new disk, initializing [ 4817.219533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4817.274303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4817.290064] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4822.528143] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4833.153413] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4839.320442] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4845.572528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4845.809661] Lustre: lustre-OST0000: new disk, initializing [ 4845.813838] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4845.820345] Lustre: 151301:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4845.910878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4851.854399] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4855.883178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4855.899502] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4856.019764] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4860.419317] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4864.416276] Lustre: Failing over lustre-OST0000 [ 4864.558145] Lustre: server umount lustre-OST0000 complete [ 4882.816081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4882.848164] Lustre: 147558:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784048615/real 1784048615] req@000000001b6f05b8 x1870710447308416/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1784048631 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4882.885388] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4883.109201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4883.143754] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4888.289578] Lustre: 147556:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784048620/real 1784048620] req@00000000b177f4b6 x1870710447310208/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1784048636 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4888.326917] Lustre: 147556:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4888.560673] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4889.254496] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4890.081158] Lustre: 147557:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784048620/real 1784048620] req@00000000fbc3389d x1870710447310336/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1784048636 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4892.856759] Lustre: lustre-OST0000: Recovery over after 0:04, of 3 clients 3 recovered and 0 were evicted. [ 4892.862580] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4893.211661] Lustre: 147557:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784048625/real 1784048625] req@00000000dd554bea x1870710447311488/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1784048641 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4893.212751] Lustre: Failing over lustre-MDT0000 [ 4893.515728] Lustre: server umount lustre-MDT0000 complete [ 4893.665630] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4893.668567] LustreError: 149340:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4893.675763] Lustre: Skipped 3 previous similar messages [ 4893.694113] LustreError: 149340:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4898.010968] LustreError: 149338:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4898.337128] Lustre: 147557:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784048630/real 1784048630] req@0000000095e9b75c x1870710447312512/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1784048646 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4898.368116] Lustre: 147557:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4903.076449] LustreError: 149340:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4903.111852] LustreError: 149340:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4908.193636] LustreError: 149338:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4908.209977] LustreError: 149338:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4911.501640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4911.645499] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4911.829413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4911.879306] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4913.315230] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4915.898745] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4917.219950] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4917.235692] Lustre: Skipped 1 previous similar message [ 4917.254606] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4917.300283] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4922.352893] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4922.358392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4922.368942] Lustre: Skipped 1 previous similar message [ 4922.380596] Lustre: Skipped 1 previous similar message [ 4924.483754] Lustre: server umount lustre-OST0000 complete [ 4932.577319] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4932.582973] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4932.599466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4932.605739] Lustre: Skipped 1 previous similar message [ 4934.439124] Lustre: server umount lustre-MDT0000 complete [ 4937.281167] LustreError: 150127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784048685 with bad export cookie 12625471559923281623 [ 4937.283345] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4937.286322] LustreError: 150127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4937.538789] Lustre: server umount lustre-MDT0001 complete [ 4945.437966] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 4948.063560] Key type lgssc unregistered [ 4948.321764] LNet: 154953:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4948.330220] LNetError: 154953:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4948.349282] LNet: Removed LNI 192.168.202.154@tcp [ 4949.573907] Key type .llcrypt unregistered [ 4949.575506] Key type ._llcrypt unregistered [ 4988.504765] Key type ._llcrypt registered [ 4988.515223] Key type .llcrypt registered [ 4988.638483] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 4990.286353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4990.329451] alg: No test for adler32 (adler32-zlib) [ 4991.456394] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 4991.690628] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 4993.384289] Key type lgssc registered [ 4994.444129] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5002.399319] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5002.437442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5003.821557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5008.131388] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 5013.989412] LustreError: 156478:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5014.000756] LustreError: 156478:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5018.398646] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5019.113545] LustreError: 156479:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5022.922462] LustreError: 156507:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5022.931559] LustreError: 156507:0:(lod_sub_object.c:937:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5022.939801] LustreError: 156507:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 5023.500507] Lustre: server umount lustre-MDT0000 complete [ 5039.817625] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 5052.287342] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 5090.925896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5091.129434] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5091.150527] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5091.223496] Lustre: lustre-MDT0000: new disk, initializing [ 5091.301567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5091.331305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5095.606068] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 5106.921647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5107.077263] Lustre: 159812:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5107.138986] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5107.144721] Lustre: Skipped 1 previous similar message [ 5107.226875] Lustre: lustre-MDT0001: new disk, initializing [ 5107.317679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5107.355389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5107.370851] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5112.066819] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 5122.428422] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5129.500727] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5137.562408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5137.791793] Lustre: lustre-OST0000: new disk, initializing [ 5137.795530] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5137.803171] Lustre: 161778:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5137.872685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5144.131199] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 5145.698303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5145.733186] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5145.830402] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5154.215982] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5159.468736] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5161.462571] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 13:08:27 (1784048907) [ 5409.252936] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5409.258303] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5409.270631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5412.323165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5412.323740] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5412.348413] Lustre: Skipped 1 previous similar message [ 5412.368376] Lustre: Skipped 1 previous similar message [ 5414.658534] Lustre: server umount lustre-MDT0000 complete [ 5417.467129] LustreError: 159822:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5417.503791] LustreError: 159822:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5418.390090] LustreError: 159802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784049166 with bad export cookie 7127865096512846587 [ 5418.390879] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5418.403117] LustreError: 159802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5422.560734] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5422.565114] LustreError: 159817:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5422.567365] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5422.567372] Lustre: Skipped 1 previous similar message [ 5424.735633] Lustre: server umount lustre-MDT0001 complete [ 5443.040185] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5443.212680] Lustre: server umount lustre-OST0000 complete [ 5453.825805] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 5464.594205] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 5505.950852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5506.149182] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5506.180937] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5506.271784] Lustre: lustre-MDT0000: new disk, initializing [ 5506.374189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5506.407062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5511.154959] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 5520.197349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5520.289785] Lustre: 166441:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5520.330809] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5520.333601] Lustre: Skipped 1 previous similar message [ 5520.404971] Lustre: lustre-MDT0001: new disk, initializing [ 5520.581982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5520.631830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5520.649376] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5524.966704] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 5533.221240] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5538.834103] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5544.145608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5544.443881] Lustre: lustre-OST0000: new disk, initializing [ 5544.450125] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5544.456518] Lustre: 168407:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5544.580733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5550.868186] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 5553.204985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5553.213569] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5553.341651] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5560.216267] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5564.706472] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5564.957416] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5568.956415] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5569.247862] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5573.601220] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5573.607102] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5573.624783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5573.632063] Lustre: Skipped 1 previous similar message [ 5579.247609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5579.267473] Lustre: Skipped 3 previous similar messages [ 5585.376157] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5585.616994] Lustre: server umount lustre-OST0000 complete [ 5589.477922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5589.492108] Lustre: Skipped 1 previous similar message [ 5589.498510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5589.504734] Lustre: Skipped 3 previous similar messages [ 5595.966811] Lustre: server umount lustre-MDT0000 complete [ 5599.714866] LustreError: 166454:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5599.733352] LustreError: 166454:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5599.914583] LustreError: 166433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784049348 with bad export cookie 7127865096513005417 [ 5599.932853] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5600.203925] Lustre: server umount lustre-MDT0001 complete [ 5608.400910] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 13:15:54 (1784049354) [ 5626.505562] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 5635.484401] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 5666.227709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5666.779858] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5666.832891] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5666.987731] Lustre: lustre-MDT0000: new disk, initializing [ 5667.182894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5667.217191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5672.741397] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 5685.401642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5685.621424] Lustre: 173164:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5685.855248] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5690.611799] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 5699.317630] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5705.621995] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5714.063444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5714.419283] Lustre: lustre-OST0000: new disk, initializing [ 5714.422604] Lustre: Skipped 1 previous similar message [ 5714.436122] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5714.444758] Lustre: Skipped 2 previous similar messages [ 5714.452291] Lustre: 175141:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5721.299356] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 5724.805202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5724.819301] Lustre: Skipped 1 previous similar message [ 5724.842245] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5724.987617] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5730.953542] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5740.516045] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5740.518198] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5740.534431] LustreError: Skipped 2 previous similar messages [ 5740.535225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5740.535230] Lustre: Skipped 2 previous similar messages [ 5740.559956] Lustre: Skipped 2 previous similar messages [ 5744.381738] Lustre: server umount lustre-OST0000 complete [ 5754.717201] Lustre: server umount lustre-MDT0000 complete [ 5755.873681] LustreError: 173171:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5755.886167] LustreError: 173171:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5758.994250] LustreError: 173154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784049507 with bad export cookie 7127865096513006299 [ 5759.002691] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5759.009578] LustreError: 173154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5759.376658] Lustre: server umount lustre-MDT0001 complete [ 5768.114641] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 13:18:34 (1784049514) [ 5785.151613] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 5797.094569] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 5838.720957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5839.140428] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5839.186727] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5839.326388] Lustre: lustre-MDT0000: new disk, initializing [ 5839.463812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5839.469488] Lustre: Skipped 2 previous similar messages [ 5839.513483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5845.733604] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 5857.186058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5857.292058] Lustre: 179466:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5857.594466] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5863.448759] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 5873.142594] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5879.084731] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5886.094103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5886.346537] Lustre: 181443:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5892.747687] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 5895.331954] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5895.430139] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5903.713761] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5916.142662] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5916.150210] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5916.180788] Lustre: Skipped 2 previous similar messages [ 5916.192481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5916.196575] Lustre: Skipped 3 previous similar messages [ 5918.033547] Lustre: server umount lustre-OST0000 complete [ 5924.323399] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5924.331105] LustreError: Skipped 1 previous similar message [ 5928.205491] Lustre: server umount lustre-MDT0000 complete [ 5931.497590] LustreError: 179477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5931.530531] LustreError: 179477:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5931.989201] LustreError: 179458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784049680 with bad export cookie 7127865096513007377 [ 5931.992349] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5932.000292] LustreError: 179458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5932.860407] Lustre: server umount lustre-MDT0001 complete [ 5944.919866] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 5947.554441] Key type lgssc unregistered [ 5947.807375] LNet: 183299:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5947.822557] LNetError: 183299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5947.847774] LNet: Removed LNI 192.168.202.154@tcp [ 5948.520804] Key type .llcrypt unregistered [ 5948.522531] Key type ._llcrypt unregistered [ 5979.242469] Key type ._llcrypt registered [ 5979.244951] Key type .llcrypt registered [ 5979.362146] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 5980.881884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5980.927184] alg: No test for adler32 (adler32-zlib) [ 5982.077238] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 5982.424583] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 5984.160177] Key type lgssc registered [ 5985.513123] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5994.237595] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5994.252063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5995.716505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6001.209827] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6006.242857] LustreError: 184717:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6006.281774] LustreError: 184717:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6011.387707] LustreError: 184718:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6013.809849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6014.481550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6023.815659] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6033.104973] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6040.061172] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6049.687803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6050.073710] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6058.761691] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6065.486732] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6067.659704] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6090.214352] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6090.237348] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6090.254583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6091.239027] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6091.244114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6091.245641] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6091.284470] Lustre: Skipped 1 previous similar message [ 6092.833563] Lustre: server umount lustre-OST0000 complete [ 6101.482260] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6101.483504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6101.509439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6101.514245] Lustre: Skipped 1 previous similar message [ 6102.941361] Lustre: server umount lustre-MDT0000 complete [ 6106.601370] LustreError: 184711:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6106.623212] LustreError: 184711:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6106.724537] LustreError: 186562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784049855 with bad export cookie 6556810142262282311 [ 6106.746367] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6107.039930] Lustre: server umount lustre-MDT0001 complete [ 6121.697561] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 6126.745435] Key type lgssc unregistered [ 6127.178724] LNet: 188272:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6127.191485] LNetError: 188272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6127.213237] LNet: Removed LNI 192.168.202.154@tcp [ 6128.654275] Key type .llcrypt unregistered [ 6128.658167] Key type ._llcrypt unregistered [ 6164.868802] Key type ._llcrypt registered [ 6164.874091] Key type .llcrypt registered [ 6165.083332] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 6166.335665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6166.453807] alg: No test for adler32 (adler32-zlib) [ 6167.586940] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 6167.763107] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 6169.427467] Key type lgssc registered [ 6171.036441] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6180.286278] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6180.306938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6182.060472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6186.930560] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6191.589824] LustreError: 189698:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6191.611245] LustreError: 189698:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6196.707321] LustreError: 189699:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6197.690757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6198.211290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6202.972810] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6210.822414] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6217.887834] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6225.084707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6225.376813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6227.472064] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6231.500419] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6239.625683] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6252.728931] Lustre: lustre-OST0000: Not available for connect from 192.168.202.54@tcp (stopping) [ 6253.027537] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6253.044615] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6253.073241] Lustre: Skipped 1 previous similar message [ 6257.824330] Lustre: lustre-OST0000: Not available for connect from 192.168.202.54@tcp (stopping) [ 6257.828254] Lustre: Skipped 2 previous similar messages [ 6262.971115] Lustre: lustre-OST0000: Not available for connect from 192.168.202.54@tcp (stopping) [ 6262.984727] Lustre: Skipped 2 previous similar messages [ 6264.288593] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6264.697793] Lustre: server umount lustre-OST0000 complete [ 6273.514629] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8bc7d157a000.ost_server_uuid 50 [ 6275.156811] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6290.404889] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6290.416056] LustreError: Skipped 1 previous similar message [ 6290.419895] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6290.438658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6290.456573] Lustre: Skipped 2 previous similar messages [ 6292.516276] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6295.537732] Lustre: server umount lustre-MDT0000 complete [ 6297.581592] LustreError: 190874:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6297.615918] LustreError: 190874:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6299.978402] LustreError: 189681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784050048 with bad export cookie 11432784015903498285 [ 6299.987156] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6299.998703] LustreError: 189681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6300.375411] Lustre: server umount lustre-MDT0001 complete [ 6309.769350] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 13:27:35 (1784050055) [ 6317.145941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6317.625391] LustreError: 193616:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6317.669086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6322.585659] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6322.658988] LustreError: 193617:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6331.435778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6331.833510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6336.643813] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6344.573959] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6350.762847] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6358.124563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6358.531489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6362.608925] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6367.339539] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6378.745450] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6385.945248] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6386.214215] Lustre: lustre-OST0001: new disk, initializing [ 6386.217310] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6386.222322] Lustre: 196459:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6386.313318] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6393.226447] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6393.476693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6393.495395] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6393.632507] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6402.649382] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6407.650564] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6407.657360] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6407.692890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6407.705882] Lustre: Skipped 2 previous similar messages [ 6411.775463] Lustre: server umount lustre-OST0000 complete [ 6413.497764] LustreError: 195438:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6413.553241] LustreError: 195438:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6418.941628] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6419.513907] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6425.481859] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6425.776082] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6429.671828] LustreError: 195440:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6429.732166] LustreError: 195440:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6445.029890] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6445.037677] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6445.051182] Lustre: Skipped 1 previous similar message [ 6445.061222] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6445.064987] LustreError: Skipped 1 previous similar message [ 6445.102466] Lustre: Skipped 5 previous similar messages [ 6446.259300] Lustre: server umount lustre-OST0001 complete [ 6449.638516] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6455.887631] Lustre: server umount lustre-MDT0000 complete [ 6459.181827] LustreError: 193598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784050207 with bad export cookie 11432784015903499356 [ 6459.197359] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6459.201594] LustreError: 193598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6459.699855] Lustre: server umount lustre-MDT0001 complete [ 6485.683530] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 13:30:31 (1784050231) [ 6492.850426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6493.406358] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6493.492621] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6493.797870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6498.419349] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6509.925925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6510.065354] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6510.089913] Lustre: 199900:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6514.950681] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6523.181221] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6529.928868] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6537.982451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6538.248159] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6544.637655] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6546.088220] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6552.250951] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6559.268164] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6559.357542] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6559.628630] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6559.637296] Lustre: Skipped 2 previous similar messages [ 6566.560917] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6566.592457] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6574.544355] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6580.411746] Lustre: lustre-OST0000: Not available for connect from 192.168.202.54@tcp (stopping) [ 6580.431303] Lustre: Skipped 5 previous similar messages [ 6581.739768] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6581.752841] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6581.768059] Lustre: Skipped 3 previous similar messages [ 6583.875147] Lustre: server umount lustre-OST0000 complete [ 6585.523267] LustreError: 201751:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6585.544140] LustreError: 201751:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 6602.721556] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6602.722527] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6602.745631] LustreError: Skipped 2 previous similar messages [ 6602.780631] Lustre: Skipped 2 previous similar messages [ 6606.670879] Lustre: server umount lustre-OST0001 complete [ 6612.972383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6612.994638] Lustre: Skipped 6 previous similar messages [ 6625.250627] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6625.649347] Lustre: server umount lustre-MDT0000 complete [ 6630.573917] LustreError: 201782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784050379 with bad export cookie 11432784015903500763 [ 6630.577301] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6630.583806] LustreError: 201782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6631.110099] Lustre: server umount lustre-MDT0001 complete [ 6659.045456] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 13:33:25 (1784050405) [ 6679.979414] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 6694.095812] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 6730.927893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6731.227690] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6731.264536] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6731.382871] Lustre: lustre-MDT0000: new disk, initializing [ 6731.551528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6731.601907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6737.590686] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6751.022184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6751.111688] Lustre: 208019:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6751.169987] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6751.176503] Lustre: Skipped 1 previous similar message [ 6751.241325] Lustre: lustre-MDT0001: new disk, initializing [ 6751.341424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6751.356696] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6756.585663] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6766.784673] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6773.638635] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6781.238331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6781.480241] Lustre: lustre-OST0000: new disk, initializing [ 6781.483672] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6781.488993] Lustre: 209986:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6787.162903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6787.169548] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6787.313264] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6788.859600] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6799.969133] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6805.275384] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6805.583741] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6811.052435] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6811.502902] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6818.274459] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6818.279331] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6818.284562] Lustre: Skipped 2 previous similar messages [ 6818.295835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6818.295925] LustreError: Skipped 1 previous similar message [ 6818.304323] Lustre: Skipped 5 previous similar messages [ 6820.373281] Lustre: server umount lustre-OST0000 complete [ 6839.264514] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6839.517159] Lustre: server umount lustre-MDT0000 complete [ 6842.849171] LustreError: 208030:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6842.879898] LustreError: 208030:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 6843.437843] LustreError: 209985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784050592 with bad export cookie 11432784015903502548 [ 6843.438167] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6843.455214] LustreError: 209985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6843.919901] Lustre: server umount lustre-MDT0001 complete [ 6854.316059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6861.791137] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6872.296612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6877.327427] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6886.353124] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6894.153524] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6904.573795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6910.573397] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6919.603685] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6925.171614] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6926.575521] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 6933.012215] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6933.410902] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6936.050640] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6936.058117] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6936.075698] Lustre: Skipped 3 previous similar messages [ 6936.108110] LustreError: Skipped 1 previous similar message [ 6950.368786] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6950.601097] Lustre: server umount lustre-OST0000 complete [ 6957.300950] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6962.104508] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6991.037982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6991.520892] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6991.528598] Lustre: Skipped 5 previous similar messages [ 6997.844668] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7000.752402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7000.768644] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7000.771211] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7000.787674] Lustre: Skipped 1 previous similar message [ 7005.724101] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7009.830639] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 7021.543018] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7021.549195] Lustre: Skipped 13 previous similar messages [ 7031.264906] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7031.476754] Lustre: server umount lustre-OST0000 complete [ 7042.737701] Lustre: server umount lustre-MDT0000 complete [ 7047.112745] LustreError: 212188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784050795 with bad export cookie 11432784015903503430 [ 7047.128364] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7047.595167] Lustre: server umount lustre-MDT0001 complete [ 7058.313450] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 13:40:04 (1784050804) [ 7065.553884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7066.300897] LustreError: 218194:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7066.328844] LustreError: 218194:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7072.469960] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7081.595686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7088.080538] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7098.908962] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7107.992259] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7116.393856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7124.234423] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7132.864342] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7138.390604] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7138.636571] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7144.721957] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7145.078811] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7151.982630] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7152.164227] Lustre: lustre-OST0001: new disk, initializing [ 7152.167135] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7152.173070] Lustre: 221565:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7158.663682] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7159.939942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7159.947111] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7160.107645] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7170.489856] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7177.389779] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7177.678187] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7183.449946] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7183.726775] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7191.020956] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7191.032848] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7191.034687] LustreError: Skipped 4 previous similar messages [ 7191.072387] Lustre: Skipped 6 previous similar messages [ 7192.523269] Lustre: server umount lustre-OST0001 complete [ 7199.552950] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7199.906342] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7204.997456] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7205.201941] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7235.963504] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7241.731743] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7241.778286] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7241.782086] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7243.632429] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7252.646919] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7254.453750] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7262.650331] Lustre: server umount lustre-OST0001 complete [ 7280.624540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7280.633748] Lustre: Skipped 12 previous similar messages [ 7286.957608] Lustre: server umount lustre-OST0000 complete [ 7306.208156] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7306.446091] Lustre: server umount lustre-MDT0000 complete [ 7310.241773] LustreError: 218176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784051058 with bad export cookie 11432784015903504396 [ 7310.252885] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7310.260636] LustreError: 218176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7316.738568] Lustre: server umount lustre-MDT0001 complete [ 7344.929468] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 13:44:50 (1784051090) [ 7351.785976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7352.114103] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7352.147706] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7356.569031] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7366.019617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7366.150354] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7366.214892] Lustre: 226819:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7371.384251] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7379.272718] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7385.435799] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7393.031755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7393.163961] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7402.595839] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7415.352992] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7426.794302] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7426.887560] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7433.196915] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7441.694402] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7443.483781] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7448.407919] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7448.606598] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7452.942786] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7453.124779] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7459.340717] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8bc7c7162000.ost_server_uuid 50 [ 7461.105251] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8bc7c7162000.ost_server_uuid in IDLE state after 0 sec [ 7462.204313] Lustre: Permanently deactivating lustre-OST0001 [ 7462.234057] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7466.177791] Lustre: Permanently reactivating lustre-OST0001 [ 7466.213958] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7466.221227] Lustre: Skipped 2 previous similar messages [ 7476.712309] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7476.722887] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7476.732296] LustreError: Skipped 3 previous similar messages [ 7476.769017] Lustre: Skipped 7 previous similar messages [ 7479.021337] Lustre: server umount lustre-OST0001 complete [ 7481.843140] LustreError: 228661:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7481.859078] LustreError: 228661:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 36 previous similar messages [ 7489.851866] Lustre: server umount lustre-OST0000 complete [ 7500.785063] Lustre: server umount lustre-MDT0000 complete [ 7505.165126] LustreError: 231226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784051253 with bad export cookie 11432784015903505684 [ 7505.166325] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7505.179463] LustreError: 231226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7505.774267] Lustre: server umount lustre-MDT0001 complete [ 7537.595265] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 13:48:02 (1784051282) [ 7546.713537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7546.887828] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7546.920394] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7546.931617] Lustre: Skipped 2 previous similar messages [ 7547.060422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7547.063839] Lustre: Skipped 9 previous similar messages [ 7552.473872] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7561.811420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7561.920259] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7561.976219] Lustre: 233516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7568.365395] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7577.062517] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7583.945950] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7590.853702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7591.042775] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7591.064206] Lustre: Permanently deactivating lustre-OST0000 [ 7591.076190] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7598.594611] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7600.722255] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7608.484985] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7616.356520] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7616.451519] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7624.447612] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7624.774030] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7633.969555] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7636.475304] Lustre: Permanently reactivating lustre-OST0000 [ 7636.487770] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7636.491809] Lustre: Skipped 2 previous similar messages [ 7644.830040] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0001-mdtlov_UUID (at 0@lo) in 44 seconds. I think it's dead, and I am evicting it. exp 00000000cb12a461, cur 1784051393 deadline 1784051379 last 1784051349 [ 7645.329932] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7654.808849] Lustre: server umount lustre-OST0001 complete [ 7665.604558] Lustre: server umount lustre-OST0000 complete [ 7676.007373] Lustre: server umount lustre-MDT0000 complete [ 7680.726991] LustreError: 237110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784051429 with bad export cookie 11432784015903507364 [ 7680.734980] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7680.747404] LustreError: 237110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7681.283592] Lustre: server umount lustre-MDT0001 complete [ 7694.570979] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 7697.636892] Key type lgssc unregistered [ 7697.936507] LNet: 238373:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7697.946934] LNetError: 238373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7697.967160] LNet: Removed LNI 192.168.202.154@tcp [ 7698.788199] Key type .llcrypt unregistered [ 7698.790494] Key type ._llcrypt unregistered [ 7725.335033] Key type ._llcrypt registered [ 7725.337208] Key type .llcrypt registered [ 7725.477713] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 7726.637209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7726.726180] alg: No test for adler32 (adler32-zlib) [ 7727.805211] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 7728.016276] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 7729.656419] Key type lgssc registered [ 7731.036892] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7742.265981] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7742.300257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7744.014225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7748.990306] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7754.219916] LustreError: 239842:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7754.241738] LustreError: 239842:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7758.809829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7758.948046] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7758.977808] Lustre: 239832:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7758.986578] Lustre: Permanently deactivating lustre-MDT0001 [ 7759.002821] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7759.295952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7764.889799] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7766.090104] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7773.878622] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7780.640053] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7788.046715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7788.323561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7789.354206] LustreError: 241683:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7789.396094] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7794.663081] LustreError: 241758:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7795.128712] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7799.797390] LustreError: 241758:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7803.840432] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7803.876954] LustreError: 241684:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7803.922287] LustreError: 241684:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7812.081278] LustreError: 242554:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7812.126428] LustreError: 242554:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7812.339473] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7812.703026] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7817.745657] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7817.783433] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7820.255717] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7829.837264] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7833.332812] Lustre: Permanently reactivating lustre-MDT0001 [ 7833.377913] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7833.386758] Lustre: Skipped 2 previous similar messages [ 7837.276978] Lustre: lustre-MDT0001: Client a2f18e38-c2c0-48ce-b2f5-042bb1006bfe (at 192.168.202.54@tcp) reconnecting [ 7842.419218] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7842.438588] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7842.453898] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7842.474774] LustreError: 243548:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7842.476421] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7842.485559] LustreError: 243548:0:(lod_sub_object.c:937:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7842.485679] LustreError: 243548:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.154@tcp: cfg command failed: rc = -108 [ 7842.485796] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7842.485796] [ 7842.486751] LustreError: 239835:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7844.727605] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7846.840764] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7854.581033] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7854.984844] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7860.948431] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7861.175910] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7864.385419] Lustre: Permanently deactivating lustre-MDT0001 [ 7864.428057] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7864.433921] Lustre: Skipped 2 previous similar messages [ 7871.738839] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7884.264955] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7884.266993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7884.284924] Lustre: Skipped 3 previous similar messages [ 7884.302507] Lustre: Skipped 3 previous similar messages [ 7889.381662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7889.667391] Lustre: server umount lustre-MDT0000 complete [ 7893.113927] LustreError: 239824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784051641 with bad export cookie 8055911808725270098 [ 7893.126706] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7893.156404] LustreError: 239824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7893.736775] Lustre: server umount lustre-MDT0001 complete [ 7899.633124] Lustre: server umount lustre-OST0000 complete [ 7911.825782] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 13:54:17 (1784051657) [ 7923.253394] Lustre: server umount lustre-OST0001 complete [ 7930.539467] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 7940.627536] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 7991.531860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7992.066476] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7992.080811] Lustre: Skipped 2 previous similar messages [ 7992.110161] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7992.236527] Lustre: lustre-MDT0000: new disk, initializing [ 7992.316326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7992.338360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7998.431947] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8012.692170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8012.861887] Lustre: 248129:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8012.958382] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8012.968031] Lustre: Skipped 1 previous similar message [ 8013.110622] Lustre: lustre-MDT0001: new disk, initializing [ 8013.246797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8013.312329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8013.346239] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8018.186594] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8029.711352] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8037.355215] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8045.066127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8045.378089] Lustre: lustre-OST0000: new disk, initializing [ 8045.384356] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8045.389058] Lustre: 250100:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8045.488238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8052.811893] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8054.853876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8054.871984] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8054.982862] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8063.237806] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8065.720197] LustreError: 248138:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8075.732062] LustreError: 248138:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8075.769947] LustreError: 248136:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8076.131727] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8076.417823] Lustre: lustre-OST0001: new disk, initializing [ 8076.423369] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8076.432994] Lustre: 251214:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8084.372820] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8085.662211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8085.685910] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8085.801824] LustreError: 248136:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8085.839860] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8095.821790] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8098.218589] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8101.359335] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8101.374348] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8101.374981] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8101.374985] Lustre: Skipped 3 previous similar messages [ 8106.492050] LustreError: 250082:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8106.607533] Lustre: server umount lustre-OST0001 complete [ 8126.951220] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8126.967426] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8126.975770] Lustre: Skipped 1 previous similar message [ 8126.983861] LustreError: Skipped 1 previous similar message [ 8126.985746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8126.985752] Lustre: Skipped 1 previous similar message [ 8131.674398] Lustre: server umount lustre-OST0000 complete [ 8137.189803] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8137.197273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8137.206858] Lustre: Skipped 1 previous similar message [ 8137.215456] Lustre: Skipped 1 previous similar message [ 8143.562294] Lustre: server umount lustre-MDT0000 complete [ 8147.427504] LustreError: 249245:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8147.447898] LustreError: 249245:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 8149.190866] LustreError: 248925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784051897 with bad export cookie 8055911808725272695 [ 8149.193195] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8149.215125] LustreError: 248925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8155.835783] Lustre: server umount lustre-MDT0001 complete [ 8170.057731] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 8173.345521] Key type lgssc unregistered [ 8173.750764] LNet: 253248:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8173.766534] LNetError: 253248:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8173.810146] LNet: Removed LNI 192.168.202.154@tcp [ 8174.946157] Key type .llcrypt unregistered [ 8174.947969] Key type ._llcrypt unregistered [ 8223.942705] Key type ._llcrypt registered [ 8223.944264] Key type .llcrypt registered [ 8224.033993] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 8225.606547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8225.642886] alg: No test for adler32 (adler32-zlib) [ 8226.843339] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 8227.306822] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 8228.987985] Key type lgssc registered [ 8230.590668] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8238.886885] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8238.903771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8240.141067] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8240.182350] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8240.362819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8244.496325] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8254.609726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8254.731365] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8254.787477] Lustre: 255471:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8255.313659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8260.051934] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8269.787846] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8277.880300] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8286.409964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8286.611963] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8286.883808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8293.780226] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8294.549921] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8302.833544] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8361.457809] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8361.459646] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8361.525306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8361.560156] Lustre: Skipped 1 previous similar message [ 8363.600917] Lustre: server umount lustre-OST0000 complete [ 8369.429418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8384.674908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8385.049096] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8392.859957] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8402.437732] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8402.456855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8402.456904] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8402.490860] Lustre: Skipped 1 previous similar message [ 8403.047976] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8448.485765] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8448.492529] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8448.502540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8448.530721] Lustre: Skipped 2 previous similar messages [ 8450.066783] Lustre: server umount lustre-OST0000 complete [ 8455.143067] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8455.158365] LustreError: Skipped 1 previous similar message [ 8455.167122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8455.204049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8455.227278] Lustre: Skipped 1 previous similar message [ 8458.722216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8458.726207] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8460.674108] Lustre: server umount lustre-MDT0000 complete [ 8465.800710] LustreError: 256211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784052214 with bad export cookie 999121363317173872 [ 8465.803078] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8465.825549] LustreError: 256211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8466.584381] Lustre: server umount lustre-MDT0001 complete [ 8485.369932] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 8489.976738] Key type lgssc unregistered [ 8490.286521] LNet: 260928:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8490.296126] LNetError: 260928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8490.354969] LNet: Removed LNI 192.168.202.154@tcp [ 8491.865273] Key type .llcrypt unregistered [ 8491.868758] Key type ._llcrypt unregistered [ 8524.592611] Key type ._llcrypt registered [ 8524.594752] Key type .llcrypt registered [ 8524.704397] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 8525.915714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8525.945199] alg: No test for adler32 (adler32-zlib) [ 8527.153427] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 8527.499252] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 8529.232201] Key type lgssc registered [ 8530.556696] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8539.368658] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8539.418749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8541.073468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8546.648324] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8551.398266] LustreError: 262348:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8551.424503] LustreError: 262348:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8556.514626] LustreError: 262347:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8559.952418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8560.766299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8567.019779] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8575.675420] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8582.300835] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8589.894924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8590.326784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8597.605926] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8608.054880] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8610.836458] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8641.516430] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8641.525541] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8641.530268] LustreError: Skipped 1 previous similar message [ 8641.534043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8641.543491] Lustre: Skipped 1 previous similar message [ 8644.850992] Lustre: server umount lustre-OST0000 complete [ 8651.746424] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8651.769267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8651.771435] Lustre: Skipped 1 previous similar message [ 8656.099485] Lustre: server umount lustre-MDT0000 complete [ 8656.869164] LustreError: 262348:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8656.893625] LustreError: 262348:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8660.859385] LustreError: 262327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784052409 with bad export cookie 7478994146072013178 [ 8660.869027] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8660.874524] LustreError: 262327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8661.141311] Lustre: server umount lustre-MDT0001 complete [ 8676.542421] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 8679.109419] Key type lgssc unregistered [ 8679.395019] LNet: 266654:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8679.404155] LNetError: 266654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8679.416094] LNet: Removed LNI 192.168.202.154@tcp [ 8680.262198] Key type .llcrypt unregistered [ 8680.265431] Key type ._llcrypt unregistered [ 8699.742212] Key type ._llcrypt registered [ 8699.746430] Key type .llcrypt registered [ 8699.918088] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 8701.505868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8701.557578] alg: No test for adler32 (adler32-zlib) [ 8702.751513] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 8702.993245] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 8704.672206] Key type lgssc registered [ 8705.890502] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8714.421969] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8714.461579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8716.172755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8720.826933] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8725.480025] LustreError: 267879:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8725.495934] LustreError: 267879:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8730.595398] LustreError: 267878:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8731.737304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8732.427033] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8737.636485] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8747.621831] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8756.474924] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8764.375945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8764.810859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8772.513735] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8782.342695] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8786.321230] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8801.761635] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8801.764897] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8801.778342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8801.825827] Lustre: Skipped 1 previous similar message [ 8806.903734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8806.915612] Lustre: Skipped 2 previous similar messages [ 8812.005044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8812.021066] Lustre: Skipped 1 previous similar message [ 8815.080982] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8815.322321] Lustre: server umount lustre-OST0000 complete [ 8819.681725] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8819.694601] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8819.712196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8822.242758] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8825.801238] Lustre: server umount lustre-MDT0000 complete [ 8827.364187] LustreError: 268607:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8827.376799] LustreError: 268607:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8829.940089] LustreError: 267858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784052578 with bad export cookie 3331232802313896691 [ 8829.945681] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8829.951447] LustreError: 267858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8830.367132] Lustre: server umount lustre-MDT0001 complete [ 8843.121307] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 8846.641210] Key type lgssc unregistered [ 8846.958744] LNet: 271562:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8846.965646] LNetError: 271562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8846.994977] LNet: Removed LNI 192.168.202.154@tcp [ 8847.960667] Key type .llcrypt unregistered [ 8847.969699] Key type ._llcrypt unregistered [ 8883.951886] Key type ._llcrypt registered [ 8883.954345] Key type .llcrypt registered [ 8884.054368] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 8885.668643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8885.782865] alg: No test for adler32 (adler32-zlib) [ 8886.781493] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 8887.017058] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 8888.736232] Key type lgssc registered [ 8890.088335] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8899.573785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8899.613377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8901.360894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8906.975409] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8910.837340] LustreError: 272981:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8910.859981] LustreError: 272981:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8915.947036] LustreError: 272980:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8917.560815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8918.266450] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8923.442879] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8932.185840] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8939.044380] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8947.010478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8947.681880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8955.098387] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8964.630707] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8971.263829] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8996.834291] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8996.842326] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8996.844826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8996.856272] Lustre: Skipped 1 previous similar message [ 8996.871333] Lustre: Skipped 1 previous similar message [ 9000.538446] Lustre: server umount lustre-OST0000 complete [ 9007.075465] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9007.078624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9007.112499] Lustre: Skipped 1 previous similar message [ 9012.196802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9012.210198] Lustre: Skipped 2 previous similar messages [ 9017.316666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9017.324116] Lustre: Skipped 1 previous similar message [ 9020.384649] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9020.568250] Lustre: server umount lustre-MDT0000 complete [ 9022.443856] LustreError: 272981:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9022.470456] LustreError: 272981:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9024.339524] LustreError: 272962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784052772 with bad export cookie 8860638457732669037 [ 9024.359467] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9024.653391] Lustre: server umount lustre-MDT0001 complete [ 9039.107329] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 9043.018800] Key type lgssc unregistered [ 9043.393475] LNet: 277289:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9043.407575] LNetError: 277289:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9043.428667] LNet: Removed LNI 192.168.202.154@tcp [ 9044.729889] Key type .llcrypt unregistered [ 9044.732642] Key type ._llcrypt unregistered [ 9066.234265] Key type ._llcrypt registered [ 9066.239180] Key type .llcrypt registered [ 9066.385855] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 9067.904752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9068.000883] alg: No test for adler32 (adler32-zlib) [ 9069.672967] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 9070.042513] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 9071.728794] Key type lgssc registered [ 9073.606953] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9084.832317] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9084.878943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9086.577714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9091.755212] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9097.188292] LustreError: 278528:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9097.213486] LustreError: 278528:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9101.938407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9102.410665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9108.046743] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9117.665936] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9124.077563] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9132.156252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9132.621047] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9139.356862] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9147.720828] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9157.250510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 9172.961272] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9172.965450] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9172.980181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9173.003294] Lustre: Skipped 1 previous similar message [ 9177.355883] Lustre: server umount lustre-OST0000 complete [ 9183.202682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9183.213973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9183.216132] Lustre: Skipped 1 previous similar message [ 9183.233050] Lustre: Skipped 2 previous similar messages [ 9188.036588] Lustre: server umount lustre-MDT0000 complete [ 9188.327945] LustreError: 278528:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9188.363769] LustreError: 278528:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9192.356416] LustreError: 278494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784052940 with bad export cookie 2473109516978166081 [ 9192.366110] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9192.371291] LustreError: 278494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9192.859832] Lustre: server umount lustre-MDT0001 complete [ 9205.703104] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 9208.533389] Key type lgssc unregistered [ 9208.858739] LNet: 282251:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9208.863595] LNetError: 282251:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9208.872948] LNet: Removed LNI 192.168.202.154@tcp [ 9209.899694] Key type .llcrypt unregistered [ 9209.905262] Key type ._llcrypt unregistered [ 9224.671316] Key type ._llcrypt registered [ 9224.674110] Key type .llcrypt registered [ 9224.808177] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9244.711711] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 9259.681447] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 9260.727073] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9260.885873] alg: No test for adler32 (adler32-zlib) [ 9262.013623] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 9262.325049] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 9264.000309] Key type lgssc registered [ 9265.426638] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9307.898926] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9307.920815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9309.143855] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9309.170508] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9309.260572] Lustre: lustre-MDT0000: new disk, initializing [ 9309.348444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9309.374200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9314.407829] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9326.065391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9326.223529] Lustre: 285802:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9326.278895] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9326.283473] Lustre: Skipped 1 previous similar message [ 9326.375198] Lustre: lustre-MDT0001: new disk, initializing [ 9326.587710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9326.653348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9326.685393] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9332.252428] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9342.715369] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9349.957305] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9358.164406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9358.589892] Lustre: lustre-OST0000: new disk, initializing [ 9358.597107] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9358.605873] Lustre: 287772:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9358.752047] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9365.089772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9365.108894] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9365.290518] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9365.596670] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9377.762702] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9385.464109] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9385.842752] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9391.265805] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9391.443498] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9396.194549] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9396.200457] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9396.247832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9396.262253] Lustre: Skipped 1 previous similar message [ 9400.198134] Lustre: server umount lustre-OST0000 complete [ 9406.433788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9406.440104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9406.442823] Lustre: Skipped 1 previous similar message [ 9406.448164] Lustre: Skipped 2 previous similar messages [ 9411.556666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9411.566654] Lustre: Skipped 1 previous similar message [ 9415.651191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9415.656369] Lustre: Skipped 1 previous similar message [ 9420.256142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9420.514468] Lustre: server umount lustre-MDT0000 complete [ 9424.475393] LustreError: 285795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784053173 with bad export cookie 2024893672570271121 [ 9424.481064] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9424.498249] LustreError: 285795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9424.814834] Lustre: server umount lustre-MDT0001 complete [ 9433.987112] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 14:19:39 (1784053179) [ 9441.472617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9441.897305] LustreError: 290189:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9441.922083] LustreError: 290189:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9442.023982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9447.368583] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9447.393390] LustreError: 290190:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9447.414554] LustreError: 290190:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9451.497485] LustreError: 290189:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9456.642200] LustreError: 290190:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9458.526482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9459.040935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9464.230956] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9473.771626] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9481.439638] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9489.573665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9489.940888] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9497.354471] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9508.703871] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9524.712833] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9524.730689] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9524.754445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9526.753345] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9526.771035] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9530.686563] Lustre: server umount lustre-OST0000 complete [ 9535.785342] Lustre: server umount lustre-MDT0000 complete [ 9535.977950] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9535.996102] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9536.018205] LustreError: 290190:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9540.345866] LustreError: 290169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784053288 with bad export cookie 2024893672570272003 [ 9540.356193] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9541.095145] Lustre: server umount lustre-MDT0001 complete [ 9553.422459] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 9556.496749] Key type lgssc unregistered [ 9556.917759] LNet: 293755:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9556.924682] LNetError: 293755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9557.993047] LNet: Removed LNI 192.168.202.154@tcp [ 9559.231772] Key type .llcrypt unregistered [ 9559.233617] Key type ._llcrypt unregistered [ 9587.868177] Key type ._llcrypt registered [ 9587.870974] Key type .llcrypt registered [ 9587.978218] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 9604.235777] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 9605.792465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9605.852648] alg: No test for adler32 (adler32-zlib) [ 9607.074812] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 9607.332054] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 9609.024643] Key type lgssc registered [ 9610.167299] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9663.183414] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9663.227050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9664.515495] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9664.566474] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9664.690727] Lustre: lustre-MDT0000: new disk, initializing [ 9664.810971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9664.837255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9669.889414] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9683.362197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9683.531990] Lustre: 297586:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9683.598690] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9683.606234] Lustre: Skipped 1 previous similar message [ 9683.706417] Lustre: lustre-MDT0001: new disk, initializing [ 9683.870853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9683.905816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9683.936031] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9689.568473] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9701.573083] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9708.866384] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9717.789038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9718.216831] Lustre: lustre-OST2710: new disk, initializing [ 9718.220640] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9718.228933] Lustre: 299555:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9718.375467] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9725.096756] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9727.069837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9727.083663] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9727.197922] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9736.325393] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9743.759145] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9743.931950] Lustre: lustre-OST03e8: new disk, initializing [ 9743.935618] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9743.941665] Lustre: 300618:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9744.049675] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9751.368358] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9751.703967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9751.714657] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9751.798807] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9761.649775] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9768.806952] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9769.027403] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9773.511601] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9773.681405] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9778.362473] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9778.637685] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9783.494300] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9783.769400] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9791.463875] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9791.468418] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9791.485177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9793.003827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9793.003964] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9795.864454] Lustre: server umount lustre-MDT0000 complete [ 9799.946429] LustreError: 299554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784053548 with bad export cookie 14905976607268304960 [ 9799.952535] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9799.964693] LustreError: 299554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9800.443814] Lustre: server umount lustre-MDT0001 complete [ 9815.466419] Lustre: server umount lustre-OST2710 complete [ 9819.348459] Lustre: 295382:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784053551/real 1784053551] req@000000001883b3b9 x1870715530846464/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784053567 ref 1 fl Rpc:EXNQr/200/ffffffff rc -5/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9828.775936] Lustre: server umount lustre-OST03e8 complete [ 9848.184912] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 9860.530686] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 9899.945870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9900.374250] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9900.414799] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9900.532975] Lustre: lustre-MDT0000: new disk, initializing [ 9900.645600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9900.665342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9905.401423] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9916.268751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9916.406321] Lustre: 305870:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9916.471748] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9916.482526] Lustre: Skipped 1 previous similar message [ 9916.620611] Lustre: lustre-MDT0001: new disk, initializing [ 9916.722323] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9916.748961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9916.763479] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9921.595824] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9931.680819] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9938.208417] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9945.750927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9946.170257] Lustre: lustre-OST0000: new disk, initializing [ 9946.175699] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9946.184826] Lustre: 307835:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9946.269884] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9953.468809] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9953.826260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9953.835499] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9953.988831] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9964.281481] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9970.204828] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9970.522112] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9976.477707] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9976.685520] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9979.873688] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9979.875306] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9979.894443] LustreError: Skipped 1 previous similar message [ 9979.898234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9979.898242] Lustre: Skipped 3 previous similar messages [ 9979.934550] Lustre: Skipped 3 previous similar messages [ 9984.998589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9985.007021] Lustre: Skipped 1 previous similar message [ 9993.184240] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9993.193660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9993.199395] Lustre: Skipped 3 previous similar messages [ 9993.322524] Lustre: server umount lustre-OST0000 complete [ 9998.307526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9998.328507] Lustre: Skipped 1 previous similar message [10003.425670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10003.432603] Lustre: Skipped 3 previous similar messages [10003.927602] Lustre: server umount lustre-MDT0000 complete [10007.497102] LustreError: 305863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784053756 with bad export cookie 14905976607268306647 [10007.514383] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10007.814364] Lustre: server umount lustre-MDT0001 complete [10018.348263] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 14:29:24 (1784053764) [10020.299268] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [10022.644722] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 14:29:28 (1784053768) [10026.645967] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [10054.718801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10055.065869] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10055.093139] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10055.337369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10060.901591] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10069.328357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10069.414286] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10069.448753] Lustre: 311697:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10069.519868] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10074.265688] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10097.260531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10097.499829] Lustre: Denying initial registration attempt from nid 192.168.202.154@tcp, specified as failover [10097.507144] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [10097.519352] LustreError: 313149:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [10097.538050] LustreError: 313149:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10097.544529] LustreError: 313149:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10097.607960] Lustre: server umount lustre-OST0000 complete [10097.614500] LustreError: 313149:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [10105.826810] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10105.840027] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10105.850187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10119.136923] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10119.554105] Lustre: server umount lustre-MDT0000 complete [10121.191023] LustreError: 311708:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10121.212232] LustreError: 311708:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [10123.508558] LustreError: 312437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784053872 with bad export cookie 14905976607268307536 [10123.517439] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10123.524867] LustreError: 312437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10123.962384] Lustre: server umount lustre-MDT0001 complete [10139.185930] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [10142.748715] Key type lgssc unregistered [10143.243755] LNet: 314275:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10143.253280] LNetError: 314275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10144.296761] LNet: Removed LNI 192.168.202.154@tcp [10145.333911] Key type .llcrypt unregistered [10145.376586] Key type ._llcrypt unregistered [10159.757346] Key type ._llcrypt registered [10159.761546] Key type .llcrypt registered [10159.906584] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [10161.151620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10161.184830] alg: No test for adler32 (adler32-zlib) [10162.394049] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [10162.705523] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [10164.448141] Key type lgssc registered [10165.696929] Lustre: Echo OBD driver; http://www.lustre.org/ [10222.239189] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [10232.661081] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10232.726774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10234.189660] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10234.234487] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10234.497904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10240.959560] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10251.868299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10251.980793] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10252.021791] Lustre: 317043:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10252.377940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10258.315449] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10268.901667] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10277.587182] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10298.164707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10298.483222] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10298.840432] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10306.538585] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10314.472601] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10319.329703] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10319.332427] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10319.340386] LustreError: Skipped 1 previous similar message [10319.352816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10319.374202] Lustre: Skipped 1 previous similar message [10322.970144] Lustre: server umount lustre-OST0000 complete [10329.059698] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10329.065251] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10329.092543] Lustre: Skipped 1 previous similar message [10329.097140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10334.692022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10334.707739] Lustre: Skipped 2 previous similar messages [10339.821138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10339.826397] Lustre: Skipped 1 previous similar message [10341.344804] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10341.536688] Lustre: server umount lustre-MDT0000 complete [10346.450609] LustreError: 317034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784054095 with bad export cookie 5912327504830173692 [10346.471979] LustreError: 317034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10346.478257] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10346.995243] Lustre: server umount lustre-MDT0001 complete [10359.681856] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [10362.376611] Key type lgssc unregistered [10362.689023] LNet: 320620:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10362.695127] LNetError: 320620:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10362.719047] LNet: Removed LNI 192.168.202.154@tcp [10363.643805] Key type .llcrypt unregistered [10363.647641] Key type ._llcrypt unregistered [10396.292433] Key type ._llcrypt registered [10396.297431] Key type .llcrypt registered [10396.473993] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [10398.816522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10398.841059] alg: No test for adler32 (adler32-zlib) [10400.017727] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [10400.407865] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [10402.128229] Key type lgssc registered [10403.110455] Lustre: Echo OBD driver; http://www.lustre.org/ [10410.413354] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10410.428732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10411.853808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10417.963304] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10422.242164] LustreError: 322039:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10422.274551] LustreError: 322039:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10426.984213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10427.510290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10433.603500] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10441.968617] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10449.029463] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10458.726562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10459.222188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10465.607469] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10473.947559] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10495.169885] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.54@tcp (stopping) [10496.569833] Lustre: server umount lustre-MDT0000 complete [10497.514836] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10497.539221] LustreError: 323880:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10497.542349] Lustre: Skipped 2 previous similar messages [10497.567930] LustreError: 323880:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10500.255160] LustreError: 322034:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10501.178965] LustreError: 322021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784054249 with bad export cookie 5203614061786968195 [10501.225468] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10502.625774] LustreError: 322035:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10502.650453] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10502.681133] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10505.413693] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.54@tcp (stopping) [10506.657380] LustreError: 322034:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10506.677790] LustreError: 322034:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10507.687776] Lustre: server umount lustre-MDT0001 complete [10514.707102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10524.463937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10525.793182] LustreError: 325641:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [10525.821194] LustreError: 325641:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@000000000b30c8e0 x1870716362332928/t0(0) o250->MGC192.168.202.154@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1784054274 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10525.856274] LustreError: 325641:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [10526.184558] LustreError: 321349:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d619c859 x1870716362333312/t0(0) o250->MGC192.168.202.154@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10526.642085] LustreError: 323870:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10526.822465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10526.931907] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10533.567736] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10536.045500] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10544.603318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10545.297553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10550.262153] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10550.958312] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10559.925950] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10562.035850] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10568.455946] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10570.425647] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10577.890070] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10577.895385] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10577.914116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10577.918835] Lustre: Skipped 1 previous similar message [10580.964873] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10580.967888] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10582.798402] Lustre: server umount lustre-OST0000 complete [10591.215826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10591.225984] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10591.243203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10591.252958] Lustre: Skipped 2 previous similar messages [10595.475138] Lustre: server umount lustre-MDT0000 complete [10596.321896] LustreError: 326950:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10596.335952] LustreError: 326950:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [10599.626342] LustreError: 322020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784054348 with bad export cookie 5203614061786975643 [10599.637668] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10599.638533] LustreError: 322020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10599.897989] Lustre: server umount lustre-MDT0001 complete [10612.184514] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [10615.168640] Key type lgssc unregistered [10615.531956] LNet: 328331:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10615.540467] LNetError: 328331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10616.550771] LNet: Removed LNI 192.168.202.154@tcp [10617.429134] Key type .llcrypt unregistered [10617.430834] Key type ._llcrypt unregistered [10646.377100] Key type ._llcrypt registered [10646.381824] Key type .llcrypt registered [10646.598174] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [10648.358490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10648.628739] alg: No test for adler32 (adler32-zlib) [10649.777949] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [10650.042736] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [10651.737161] Key type lgssc registered [10652.862203] Lustre: Echo OBD driver; http://www.lustre.org/ [10662.488704] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10662.533777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10664.039925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10669.531551] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10673.638486] LustreError: 329749:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10673.672026] LustreError: 329749:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10678.761782] LustreError: 329750:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10679.723205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10680.073399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10684.619988] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10694.311665] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10702.008369] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10709.990565] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10710.016052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10710.022795] Lustre: Skipped 1 previous similar message [10712.430850] Lustre: server umount lustre-MDT0000 complete [10715.107597] LustreError: 329750:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10715.143767] LustreError: 329750:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10716.830051] LustreError: 329729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784054465 with bad export cookie 8092043661795744701 [10716.860153] LustreError: 329729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10716.861708] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10717.221440] Lustre: server umount lustre-MDT0001 complete [10736.319162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10736.533136] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10736.568729] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10736.730766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10741.947831] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10752.678320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10752.771829] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10752.813511] Lustre: 332339:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10753.126905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10757.685440] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10766.001635] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10772.851833] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10782.730662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10783.242821] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10783.259591] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10783.272140] LustreError: 334225:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10783.294485] LustreError: 334225:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10783.312321] LustreError: 334225:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10783.486413] Lustre: server umount lustre-OST0000 complete [10783.492148] LustreError: 334225:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [10795.074607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10795.450536] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10795.816361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10802.855288] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10803.220394] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10814.145554] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10825.168971] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10825.324879] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10825.511983] Lustre: lustre-OST0001: new disk, initializing [10825.518379] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10825.526294] Lustre: 335768:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10825.695937] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10830.930457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10830.941566] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10831.129362] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10833.658323] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10845.482333] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10851.812087] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10851.815785] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10851.829884] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10851.871834] Lustre: Skipped 2 previous similar messages [10856.943509] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10856.957325] Lustre: Skipped 2 previous similar messages [10862.059784] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10862.065258] Lustre: Skipped 1 previous similar message [10863.072229] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10863.277725] Lustre: server umount lustre-OST0001 complete [10867.183566] LustreError: 334694:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10867.218289] LustreError: 334694:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10869.218705] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10869.232232] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10869.244500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10872.307888] LustreError: 336023:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10872.319409] LustreError: 336023:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10873.926368] Lustre: server umount lustre-OST0000 complete [10880.993962] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10881.006918] LustreError: Skipped 1 previous similar message [10881.014790] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10881.033430] Lustre: Skipped 1 previous similar message [10881.041066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10881.049193] Lustre: Skipped 3 previous similar messages [10885.039833] Lustre: server umount lustre-MDT0000 complete [10887.651696] LustreError: 334527:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10887.675346] LustreError: 334527:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10889.145335] LustreError: 332332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784054637 with bad export cookie 8092043661795745247 [10889.153445] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10889.613994] Lustre: server umount lustre-MDT0001 complete [10903.485628] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [10906.703693] Key type lgssc unregistered [10907.105933] LNet: 337729:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10907.122984] LNetError: 337729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10908.149540] LNet: Removed LNI 192.168.202.154@tcp [10909.267606] Key type .llcrypt unregistered [10909.269233] Key type ._llcrypt unregistered [10994.505372] Key type ._llcrypt registered [10994.506877] Key type .llcrypt registered [10994.672777] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [11015.531034] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [11018.220980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11018.284273] alg: No test for adler32 (adler32-zlib) [11019.564901] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [11019.950890] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [11021.696823] Key type lgssc registered [11023.670868] Lustre: Echo OBD driver; http://www.lustre.org/ [11070.145852] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11070.164413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11071.384516] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11071.443769] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11071.544856] Lustre: lustre-MDT0000: new disk, initializing [11071.659805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11071.699877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11076.262929] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11089.176877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11089.313469] Lustre: 342668:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11089.465144] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11089.470606] Lustre: Skipped 1 previous similar message [11089.504957] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11089.596861] Lustre: lustre-MDT0001: new disk, initializing [11089.680825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11089.714070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11089.726504] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11095.946283] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11108.971110] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11116.453768] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11124.197540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11124.552263] Lustre: lustre-OST0000: new disk, initializing [11124.556377] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11124.567443] Lustre: 344639:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11124.640951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11133.001286] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11134.100814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11134.110077] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11134.259929] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11144.726641] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11150.071758] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11150.322675] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11154.824449] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11155.071882] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11160.036285] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11160.042260] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11160.054911] LustreError: Skipped 1 previous similar message [11160.055925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11160.084131] Lustre: Skipped 1 previous similar message [11163.606709] Lustre: server umount lustre-OST0000 complete [11170.274053] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11170.303102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11170.314513] Lustre: Skipped 2 previous similar messages [11174.312632] Lustre: server umount lustre-MDT0000 complete [11178.518288] LustreError: 342659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784054927 with bad export cookie 18371527051482848721 [11178.524939] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11178.527308] LustreError: 342659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11178.848797] Lustre: server umount lustre-MDT0001 complete [11191.345697] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 14:48:56 (1784054936) [11201.126249] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 14:49:06 (1784054946) [11212.314666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11212.786439] LustreError: 347397:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11212.805298] LustreError: 347397:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11212.908759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11217.905316] LustreError: 347398:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11219.630483] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11223.021944] LustreError: 347397:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11228.136957] LustreError: 347398:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11230.329648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11230.924427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11236.138986] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11245.375164] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11252.888456] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11261.444081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11261.915247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11270.173784] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11279.774411] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11293.371123] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11295.573515] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11297.590722] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11300.012428] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11302.454155] Lustre: Failing over lustre-MDT0000 [11302.861199] Lustre: server umount lustre-MDT0000 complete [11302.888444] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11302.903622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11302.930788] Lustre: Skipped 1 previous similar message [11302.939984] LustreError: 348997:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11311.282531] LustreError: 349475:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11311.312881] LustreError: 349475:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [11320.096114] Lustre: 340891:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784055052/real 1784055052] req@00000000f1e7d3d0 x1870717012486272/t0(0) o400->MGC192.168.202.154@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1784055068 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11320.130228] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11323.463512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11329.508948] LustreError: 340889:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005f7ec9bd x1870717012491648/t0(0) o250->MGC192.168.202.154@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11329.545096] LustreError: 348997:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11329.561248] LustreError: 348997:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [11329.755146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11329.815316] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11331.752261] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11334.827337] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11335.149378] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11335.188524] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [11335.275257] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11345.629365] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11347.623001] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11351.653327] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11354.013317] Lustre: Failing over lustre-MDT0000 [11354.540977] Lustre: server umount lustre-MDT0000 complete [11355.617755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11355.621167] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11355.627814] Lustre: Skipped 4 previous similar messages [11362.461707] LustreError: 349475:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11362.479261] LustreError: 349475:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [11372.000123] Lustre: 340890:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784055104/real 1784055104] req@0000000019d11ac9 x1870717012508032/t0(0) o400->MGC192.168.202.154@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1784055120 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11372.038458] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11376.690551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11382.266233] LustreError: 340889:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007c92985b x1870717012514048/t0(0) o250->MGC192.168.202.154@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11382.848697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11382.928245] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11382.955178] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11387.897728] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11387.912491] Lustre: Skipped 2 previous similar messages [11387.937709] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [11387.986471] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11389.603957] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11402.128547] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11404.477405] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11413.473498] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11413.480163] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11413.490622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11413.530307] Lustre: Skipped 1 previous similar message [11416.744545] Lustre: server umount lustre-MDT0000 complete [11421.107041] LustreError: 347378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784055169 with bad export cookie 18371527051482851430 [11421.118415] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11421.616075] Lustre: server umount lustre-MDT0001 complete [11436.660708] Lustre: server umount lustre-OST0000 complete [11456.093872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11456.644248] LustreError: 353768:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11456.671710] LustreError: 353768:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 21 previous similar messages [11456.824184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11464.444669] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11476.231395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11482.384574] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11491.883945] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11499.199716] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11508.781106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11518.596514] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11526.909417] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11529.652979] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11537.383083] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11537.388166] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11537.396966] LustreError: Skipped 1 previous similar message [11537.402170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11537.433277] Lustre: Skipped 2 previous similar messages [11537.461103] Lustre: Skipped 3 previous similar messages [11541.989496] Lustre: server umount lustre-OST0000 complete [11548.134365] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11548.150920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11554.344529] Lustre: server umount lustre-MDT0000 complete [11559.048118] LustreError: 353749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784055307 with bad export cookie 18371527051482852179 [11559.070241] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11559.076029] LustreError: 353749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11559.521352] Lustre: server umount lustre-MDT0001 complete [11571.093866] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [11574.160140] Key type lgssc unregistered [11574.481510] LNet: 357301:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11574.490109] LNetError: 357301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11574.513894] LNet: Removed LNI 192.168.202.154@tcp [11575.626889] Key type .llcrypt unregistered [11575.639517] Key type ._llcrypt unregistered [11609.509784] Key type ._llcrypt registered [11609.513532] Key type .llcrypt registered [11609.665946] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [11611.139244] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11611.219797] alg: No test for adler32 (adler32-zlib) [11612.741979] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [11613.049824] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [11614.720253] Key type lgssc registered [11616.854450] Lustre: Echo OBD driver; http://www.lustre.org/ [11628.217951] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11628.239643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11629.576570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11633.539126] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11639.780165] LustreError: 358822:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11639.796208] LustreError: 358822:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11643.546375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11643.946970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11648.506685] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11657.230596] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11664.352201] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11671.301779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11671.814040] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11674.891586] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11678.219393] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11685.775422] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11701.467950] Lustre: server umount lustre-MDT0000 complete [11704.812217] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11704.821891] LustreError: 358821:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11704.829036] Lustre: Skipped 2 previous similar messages [11704.870798] LustreError: 358821:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11705.744175] LustreError: 358804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784055454 with bad export cookie 18096520813777884363 [11705.750578] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11705.759254] LustreError: 358804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11706.260332] Lustre: server umount lustre-MDT0001 complete [11721.958563] Lustre: server umount lustre-OST0000 complete [11738.943367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11739.538433] LustreError: 362556:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11739.608282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11744.738195] LustreError: 362557:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11744.894978] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11749.859400] LustreError: 362556:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11755.114418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11755.464509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11760.746696] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11770.348591] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11776.733483] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11783.987973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11784.308163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11785.346526] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11791.136749] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11799.602666] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11811.809980] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11811.814460] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11811.825975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11815.403867] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11815.408528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11815.434665] Lustre: Skipped 1 previous similar message [11817.888507] Lustre: server umount lustre-MDT0000 complete [11820.519048] LustreError: 362553:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11820.530682] LustreError: 362553:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11822.459905] LustreError: 364402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784055571 with bad export cookie 18096520813777885476 [11822.471378] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11822.482170] LustreError: 364402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11822.914928] Lustre: server umount lustre-MDT0001 complete [11836.730936] Lustre: server umount lustre-OST0000 complete [11853.315750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11854.025225] LustreError: 366470:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11854.083705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11860.858275] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11874.657745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11882.980543] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11895.403420] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11902.250162] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11912.226578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11912.806674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11912.811049] Lustre: Skipped 1 previous similar message [11920.828253] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11924.114713] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11930.698065] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11937.046886] Lustre: server umount lustre-OST0000 complete [11941.857205] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11941.871775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11941.897662] Lustre: Skipped 1 previous similar message [11941.906587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11944.930892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11944.943683] Lustre: Skipped 1 previous similar message [11950.052207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11950.077199] Lustre: Skipped 1 previous similar message [11951.324978] Lustre: server umount lustre-MDT0000 complete [11955.181384] LustreError: 367289:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11955.193805] LustreError: 367289:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11956.099973] LustreError: 366452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784055704 with bad export cookie 18096520813777886491 [11956.111501] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11956.115339] LustreError: 366452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11966.776332] Lustre: server umount lustre-MDT0001 complete [11981.722434] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [11984.578548] Key type lgssc unregistered [11984.998826] LNet: 370018:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11985.023983] LNetError: 370018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11986.086728] LNet: Removed LNI 192.168.202.154@tcp [11987.605987] Key type .llcrypt unregistered [11987.607599] Key type ._llcrypt unregistered [12024.826251] Key type ._llcrypt registered [12024.828218] Key type .llcrypt registered [12024.974736] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [12026.482246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12026.530528] alg: No test for adler32 (adler32-zlib) [12027.697464] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [12027.963704] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [12029.656114] Key type lgssc registered [12031.181179] Lustre: Echo OBD driver; http://www.lustre.org/ [12040.078274] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12040.092946] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12040.101692] LustreError: 371464:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [12040.119538] LustreError: 371464:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [12040.125598] LustreError: 371464:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [12040.129732] LustreError: 371464:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [12040.136729] LustreError: 371464:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [12049.297424] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [12049.311387] LustreError: 371862:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [12049.342901] LustreError: 371862:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [12049.348978] LustreError: 371862:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [12049.362276] LustreError: 371862:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [12049.368267] LustreError: 371862:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [12070.114493] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [12072.905424] Key type lgssc unregistered [12073.166230] LNet: 372865:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12073.174635] LNetError: 372865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12073.187216] LNet: Removed LNI 192.168.202.154@tcp [12073.887574] Key type .llcrypt unregistered [12073.890125] Key type ._llcrypt unregistered [12093.543252] Key type ._llcrypt registered [12093.546282] Key type .llcrypt registered [12093.670816] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [12113.172906] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [12114.804498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12114.909422] alg: No test for adler32 (adler32-zlib) [12116.369198] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [12117.061221] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [12118.978051] Key type lgssc registered [12120.581573] Lustre: Echo OBD driver; http://www.lustre.org/ [12164.005625] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12164.055946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12165.402747] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12165.425599] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12165.479997] Lustre: lustre-MDT0000: new disk, initializing [12165.536091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12165.552676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12171.934369] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12184.227047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12184.383516] Lustre: 376026:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12184.464947] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12184.468580] Lustre: Skipped 1 previous similar message [12184.560618] Lustre: lustre-MDT0001: new disk, initializing [12184.666286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12184.725801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12184.750379] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12189.890427] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12201.327070] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12208.637807] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12216.454118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12216.849278] Lustre: lustre-OST0000: new disk, initializing [12216.856279] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12216.864371] Lustre: 377993:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12216.952577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12223.985077] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12225.108561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12225.118337] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12225.272111] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12234.969730] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12240.371253] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12240.863302] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12247.253402] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12247.637629] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12251.116295] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12251.139113] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12251.166301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12256.003495] Lustre: server umount lustre-OST0000 complete [12261.345550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12261.357216] Lustre: Skipped 1 previous similar message [12261.367499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12261.376809] Lustre: Skipped 2 previous similar messages [12266.576439] Lustre: server umount lustre-MDT0000 complete [12270.472132] LustreError: 376018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784056019 with bad export cookie 3832400613935476544 [12270.491202] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12270.497151] LustreError: 376018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12270.938215] Lustre: server umount lustre-MDT0001 complete [12281.722692] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [12283.768834] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 15:07:09 (1784056029) [12290.692439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12291.106297] LustreError: 380457:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12291.120260] LustreError: 380457:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12291.168091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12296.181140] LustreError: 380458:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12296.444335] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12301.286220] LustreError: 380457:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12306.403978] LustreError: 380458:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12306.540273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12307.010353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12312.847330] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12321.695846] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12328.470104] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12336.866312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12337.360166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12346.300798] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12353.134327] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12361.972059] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12362.265375] Lustre: lustre-OST0001: new disk, initializing [12362.268256] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12362.272863] Lustre: 383297:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12362.376087] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12368.186221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12368.200810] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12368.331657] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12370.528637] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12381.754051] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12388.837099] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12388.858476] LustreError: Skipped 1 previous similar message [12388.860278] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12388.869460] Lustre: Skipped 1 previous similar message [12388.878449] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12388.880919] Lustre: Skipped 1 previous similar message [12392.285715] Lustre: server umount lustre-OST0001 complete [12393.960884] LustreError: 384106:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12393.994856] LustreError: 384106:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12404.197990] LustreError: 384106:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12404.231675] LustreError: 384106:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12412.385362] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12412.389171] LustreError: Skipped 1 previous similar message [12412.397962] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12412.416052] Lustre: Skipped 1 previous similar message [12412.420364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12412.426891] Lustre: Skipped 1 previous similar message [12414.444372] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12416.845483] Lustre: server umount lustre-OST0000 complete [12424.675853] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12424.676172] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12424.719112] Lustre: Skipped 1 previous similar message [12424.726157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12424.726166] Lustre: Skipped 2 previous similar messages [12427.089193] Lustre: server umount lustre-MDT0000 complete [12429.793320] LustreError: 381383:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12429.834372] LustreError: 381383:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [12431.402842] LustreError: 380438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784056179 with bad export cookie 3832400613935477419 [12431.427662] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12431.434539] LustreError: 380438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12438.127808] Lustre: server umount lustre-MDT0001 complete [12451.455567] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [12453.996807] Key type lgssc unregistered [12454.302905] LNet: 385290:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12454.318227] LNetError: 385290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12454.334542] LNet: Removed LNI 192.168.202.154@tcp [12455.241528] Key type .llcrypt unregistered [12455.244873] Key type ._llcrypt unregistered [12509.872465] Key type ._llcrypt registered [12509.876645] Key type .llcrypt registered [12509.957718] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [12511.181918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12511.195941] alg: No test for adler32 (adler32-zlib) [12512.482781] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [12513.028808] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [12514.922238] Key type lgssc registered [12516.074266] Lustre: Echo OBD driver; http://www.lustre.org/ [12525.427967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12525.442177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12526.722342] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12526.766651] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12526.934807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12531.886233] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12542.115944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12542.239690] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12542.313331] Lustre: 388091:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12542.717516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12548.483943] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12559.955981] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12566.994345] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12573.666433] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12573.667667] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12573.694666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12573.717980] Lustre: Skipped 1 previous similar message [12575.785610] Lustre: server umount lustre-MDT0000 complete [12580.736865] LustreError: 388082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784056329 with bad export cookie 1528536529602583949 [12580.738848] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12580.745553] LustreError: 388082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12581.185073] Lustre: server umount lustre-MDT0001 complete [12594.036268] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 15:12:19 (1784056339) [12602.092839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12602.609792] LustreError: 390603:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12602.624064] LustreError: 390603:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12602.676122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12607.973667] LustreError: 390604:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12607.992678] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12613.091800] LustreError: 390603:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12617.872629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12618.283618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12622.858719] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12631.208844] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12637.843528] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12645.116564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12645.257547] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12645.506624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12652.951200] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12661.806411] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12669.827614] Lustre: Permanently deactivating lustre-OST0000 [12669.852960] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12678.200202] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12680.642812] LustreError: 393592:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12680.649492] LustreError: 393592:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [12680.653759] LustreError: 393592:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12684.627788] LustreError: 393640:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [12684.635486] LustreError: 393640:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12693.274457] Lustre: server umount lustre-OST0000 complete [12697.779992] LustreError: 393890:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [12697.785664] LustreError: 393890:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12700.129604] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12700.146104] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12700.168509] Lustre: Skipped 1 previous similar message [12700.179458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12704.227547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12704.246779] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12709.354095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12709.374198] Lustre: Skipped 2 previous similar messages [12710.459517] Lustre: server umount lustre-MDT0000 complete [12714.466889] LustreError: 391329:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12714.496881] LustreError: 391329:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12715.418179] LustreError: 390585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784056463 with bad export cookie 1528536529602584530 [12715.424489] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12715.430770] LustreError: 390585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12725.923837] Lustre: server umount lustre-MDT0001 complete [12735.975865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12740.398378] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12743.707709] LustreError: 395050:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12750.685614] Lustre: 395390:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.154@tcp [12752.933327] Lustre: server umount MGS complete [12766.097126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12766.634956] LustreError: 396115:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12766.676437] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12766.683216] Lustre: Skipped 1 previous similar message [12766.779562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12772.224727] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12776.930563] LustreError: 396116:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12776.946101] LustreError: 396116:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12781.194596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12781.493683] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12787.507676] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12796.993592] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12804.331874] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12806.793813] Lustre: Permanently reactivating lustre-OST0000 [12806.828568] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12806.844455] Lustre: Skipped 2 previous similar messages [12833.761765] LustreError: 397846:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12843.779327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12844.051700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12844.060986] Lustre: Skipped 1 previous similar message [12851.691950] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12859.858620] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12868.578365] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12868.586625] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12868.602164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12879.877799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12879.888731] Lustre: Skipped 5 previous similar messages [12880.353330] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12880.630435] Lustre: server umount lustre-OST0000 complete [12884.963478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12884.980617] Lustre: Skipped 2 previous similar messages [12899.297778] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12899.602319] Lustre: server umount lustre-MDT0000 complete [12900.320955] LustreError: 396843:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12900.351048] LustreError: 396843:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12903.870862] LustreError: 396095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784056652 with bad export cookie 1528536529602585573 [12903.886196] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12903.890636] LustreError: 396095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12904.345853] Lustre: server umount lustre-MDT0001 complete [12917.732342] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [12921.202700] Key type lgssc unregistered [12921.543065] LNet: 400433:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12921.556891] LNetError: 400433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12922.607339] LNet: Removed LNI 192.168.202.154@tcp [12923.772646] Key type .llcrypt unregistered [12923.777115] Key type ._llcrypt unregistered [12942.408704] Key type ._llcrypt registered [12942.412293] Key type .llcrypt registered [12942.502500] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [12958.790405] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [12959.717306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12959.739076] alg: No test for adler32 (adler32-zlib) [12961.034659] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [12961.269356] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [12962.984509] Key type lgssc registered [12964.423793] Lustre: Echo OBD driver; http://www.lustre.org/ [13005.123721] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13005.158934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13006.494973] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13006.520675] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13006.590396] Lustre: lustre-MDT0000: new disk, initializing [13006.706772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13006.726992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13011.323788] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13022.944071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13023.048130] Lustre: 403592:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13023.123172] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13023.128581] Lustre: Skipped 1 previous similar message [13023.220202] Lustre: lustre-MDT0001: new disk, initializing [13023.352084] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13023.407371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13023.419720] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13028.400641] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13039.947391] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13046.710591] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13054.672813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13055.028430] Lustre: lustre-OST0000: new disk, initializing [13055.032580] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13055.044491] Lustre: 405561:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13055.149676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13061.767154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13061.795749] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13062.011058] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13062.316033] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13073.500341] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13079.105542] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13079.330823] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13084.476503] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13084.745882] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13087.715090] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13087.719348] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13087.731454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13092.851118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13092.875134] Lustre: Skipped 1 previous similar message [13093.187643] Lustre: server umount lustre-OST0000 complete [13097.953531] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13097.957207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13097.973279] Lustre: Skipped 2 previous similar messages [13097.998641] Lustre: Skipped 2 previous similar messages [13103.073552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13103.079204] Lustre: Skipped 1 previous similar message [13104.252169] Lustre: server umount lustre-MDT0000 complete [13108.397712] LustreError: 403582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784056856 with bad export cookie 13597916230735177194 [13108.407353] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13108.794727] Lustre: server umount lustre-MDT0001 complete [13117.792768] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [13119.570674] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 15:21:05 (1784056865) [13126.727371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13127.338407] LustreError: 408022:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13127.356986] LustreError: 408022:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [13127.433742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13132.381550] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13132.784099] LustreError: 408023:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13136.934240] LustreError: 408022:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13141.258809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13141.792090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13146.396545] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13154.698984] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13160.680083] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13167.905541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13168.169624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13173.647597] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13181.242462] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13183.375739] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [13187.555408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [13192.676496] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13192.694217] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13192.740079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13196.790803] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13196.803350] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13198.864819] Lustre: server umount lustre-OST0000 complete [13207.009567] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13207.034534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13207.040106] Lustre: Skipped 3 previous similar messages [13210.565816] Lustre: server umount lustre-MDT0000 complete [13212.139309] LustreError: 408022:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13212.173230] LustreError: 408022:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13215.853605] LustreError: 408004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784056964 with bad export cookie 13597916230735178076 [13215.870264] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13215.877682] LustreError: 408004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13216.364686] Lustre: server umount lustre-MDT0001 complete [13230.551459] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [13233.816796] Key type lgssc unregistered [13234.161293] LNet: 411748:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13234.168448] LNetError: 411748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13234.179924] LNet: Removed LNI 192.168.202.154@tcp [13235.124492] Key type .llcrypt unregistered [13235.135350] Key type ._llcrypt unregistered [13261.446773] Key type ._llcrypt registered [13261.448922] Key type .llcrypt registered [13261.575554] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [13261.790470] Key type .llcrypt unregistered [13261.794136] Key type ._llcrypt unregistered [13280.782345] Key type ._llcrypt registered [13280.785549] Key type .llcrypt registered [13280.875938] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [13281.859530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13281.891738] alg: No test for adler32 (adler32-zlib) [13283.084658] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [13283.434601] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [13285.176197] Key type lgssc registered [13286.678759] Lustre: Echo OBD driver; http://www.lustre.org/ [13296.644503] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13296.657317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13298.270434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13303.299318] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13308.387624] LustreError: 413968:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13308.412527] LustreError: 413968:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13313.125087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13313.376076] LustreError: 413967:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13313.410262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13318.625876] LustreError: 413968:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13318.652416] LustreError: 413968:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13319.568493] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13322.726172] LustreError: 413967:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13322.749318] LustreError: 413967:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13322.749634] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13326.384637] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13327.848771] LustreError: 413963:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13327.907257] LustreError: 413963:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13333.710675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13334.147515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13339.480534] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13349.348596] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13349.355742] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13349.372913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13349.863130] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13349.880598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13350.806261] Lustre: server umount lustre-OST0000 complete [13354.977587] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13354.996875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13356.014641] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13357.121086] Lustre: server umount lustre-MDT0000 complete [13360.112305] LustreError: 413962:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13360.148131] LustreError: 413962:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [13361.088163] LustreError: 413947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784057109 with bad export cookie 4975796945070057659 [13361.098455] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13361.104456] LustreError: 413947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13361.459382] Lustre: server umount lustre-MDT0001 complete [13374.030931] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [13376.308704] Key type lgssc unregistered [13376.551693] LNet: 417113:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13376.557534] LNetError: 417113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13376.579587] LNet: Removed LNI 192.168.202.154@tcp [13377.723756] Key type .llcrypt unregistered [13377.725731] Key type ._llcrypt unregistered [13403.527284] Key type ._llcrypt registered [13403.531191] Key type .llcrypt registered [13403.661906] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [13405.295612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13405.403427] alg: No test for adler32 (adler32-zlib) [13406.622387] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [13406.957408] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [13408.688206] Key type lgssc registered [13409.918659] Lustre: Echo OBD driver; http://www.lustre.org/ [13417.784741] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13417.826870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13433.440981] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13438.560439] LustreError: 418503:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.154@tcp: failed processing log, type 4: rc = -110 [13465.184475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13470.774096] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13478.687713] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13486.890337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13487.501408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13487.618152] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13492.532721] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13492.723397] LustreError: 418529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13492.746155] LustreError: 418529:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13497.826493] LustreError: 418530:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13499.889258] LustreError: 419541:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13499.918429] LustreError: 419541:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13500.962978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13501.503224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13506.403891] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13514.576370] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13520.744890] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13530.745464] Lustre: server umount lustre-OST0000 complete [13535.209519] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13535.217150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13535.233987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13537.252416] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13537.255353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13537.293336] Lustre: Skipped 1 previous similar message [13542.369898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13542.376744] Lustre: Skipped 1 previous similar message [13545.537348] Lustre: server umount lustre-MDT0000 complete [13547.489120] Lustre: 417839:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784057280/real 1784057280] req@000000006c46d670 x1870719515393152/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1784057296 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13547.490266] LustreError: 419544:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13547.519657] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13547.554786] LustreError: 419544:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13548.515605] Lustre: 417840:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784057280/real 1784057280] req@00000000f8b6e45b x1870719515392640/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1784057296 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13549.634959] LustreError: 418511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784057298 with bad export cookie 7378543985831760063 [13549.653194] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13550.242763] Lustre: server umount lustre-MDT0001 complete [13561.753611] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [13564.892625] Key type lgssc unregistered [13565.162794] LNet: 422072:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13565.171180] LNetError: 422072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13565.198460] LNet: Removed LNI 192.168.202.154@tcp [13565.891327] Key type .llcrypt unregistered [13565.894590] Key type ._llcrypt unregistered [13591.373567] Key type ._llcrypt registered [13591.378510] Key type .llcrypt registered [13591.587388] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [13592.970346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13593.073698] alg: No test for adler32 (adler32-zlib) [13594.310135] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [13594.598986] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [13596.248248] Key type lgssc registered [13597.354221] Lustre: Echo OBD driver; http://www.lustre.org/ [13605.221497] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13605.257220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13606.945477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13612.391898] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13617.121837] LustreError: 423489:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13617.153657] LustreError: 423489:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13622.176799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13622.249051] LustreError: 423490:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13622.876518] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13628.430753] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13637.127666] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13643.914656] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13651.228917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13651.607862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13652.656315] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13656.888759] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13664.473715] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13672.145131] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.54@tcp (stopping) [13673.453061] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13673.454676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13673.473682] Lustre: Skipped 2 previous similar messages [13673.484265] Lustre: Skipped 2 previous similar messages [13675.436465] Lustre: server umount lustre-MDT0000 complete [13677.232292] LustreError: 424598:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13678.042056] LustreError: 423472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.54@tcp arrived at 1784057426 with bad export cookie 6145178252786045888 [13678.086649] LustreError: 423472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13680.122788] LustreError: 423470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784057428 with bad export cookie 6145178252786044999 [13680.131543] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13680.133037] LustreError: 423470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13683.683060] LustreError: 423489:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13683.684701] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13683.699355] LustreError: 423489:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13683.719487] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13683.728923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13685.734225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13686.339970] Lustre: server umount lustre-OST0000 complete [13689.833350] LustreError: 423486:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13689.852209] LustreError: 423486:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [13693.312345] Lustre: server umount lustre-MDT0001 complete [13704.376430] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [13707.795655] Key type lgssc unregistered [13708.042351] LNet: 427174:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13708.048226] LNetError: 427174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13709.099561] LNet: Removed LNI 192.168.202.154@tcp [13709.936957] Key type .llcrypt unregistered [13709.942214] Key type ._llcrypt unregistered [13735.558238] Key type ._llcrypt registered [13735.561688] Key type .llcrypt registered [13735.673405] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [13737.524375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13737.584958] alg: No test for adler32 (adler32-zlib) [13738.668791] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [13738.893487] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [13740.568206] Key type lgssc registered [13741.713293] Lustre: Echo OBD driver; http://www.lustre.org/ [13749.676231] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13749.694844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13751.372275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13756.501692] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13760.482792] LustreError: 428591:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13760.507513] LustreError: 428591:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13765.606659] LustreError: 428592:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13766.490121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13766.898466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13771.828467] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13780.313316] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13786.661453] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13793.942888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13794.284233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13796.350806] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13801.264714] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13808.495501] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13816.502634] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.54@tcp (stopping) [13816.802586] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13819.771965] Lustre: server umount lustre-MDT0001 complete [13821.605578] LustreError: 428588:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13827.048767] LustreError: 428587:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13827.083885] LustreError: 428587:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [13832.162524] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13832.168187] LustreError: 428592:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13832.183546] Lustre: Skipped 1 previous similar message [13832.189358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13832.189365] Lustre: Skipped 2 previous similar messages [13834.549714] Lustre: server umount lustre-OST0000 complete [13838.589569] Lustre: server umount lustre-MDT0000 complete [13855.316706] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [13859.373259] Key type lgssc unregistered [13859.799639] LNet: 432278:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13859.809461] LNetError: 432278:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13859.833566] LNet: Removed LNI 192.168.202.154@tcp [13860.847519] Key type .llcrypt unregistered [13860.849804] Key type ._llcrypt unregistered [13896.980621] Key type ._llcrypt registered [13896.983321] Key type .llcrypt registered [13897.094946] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [13918.765630] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [13920.522588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13920.562684] alg: No test for adler32 (adler32-zlib) [13922.067475] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [13922.444308] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [13924.168192] Key type lgssc registered [13925.806908] Lustre: Echo OBD driver; http://www.lustre.org/ [13967.461833] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [13987.288053] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13987.340762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13988.892447] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13988.965297] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13989.251944] Lustre: lustre-MDT0000: new disk, initializing [13989.446447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13989.492969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13995.640960] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [14014.091316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14014.508933] Lustre: lustre-OST0000: new disk, initializing [14014.517534] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14014.535884] Lustre: Skipped 1 previous similar message [14014.556236] Lustre: 437334:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14014.688254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14021.473605] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [14021.720868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14021.738634] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14021.799436] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14032.151643] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14049.281446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14049.492970] Lustre: 436317:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14049.616618] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14049.747753] Lustre: lustre-MDT0001: new disk, initializing [14049.933574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14049.989631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [14050.002921] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [14054.796924] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [14065.648747] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [14065.886466] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [14071.040050] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [14071.385222] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [14081.205397] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.54@tcp (stopping) [14082.023428] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [14082.024211] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14082.031413] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14082.071601] Lustre: Skipped 1 previous similar message [14083.966696] Lustre: server umount lustre-MDT0001 complete [14092.273409] LustreError: 436327:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14092.292296] LustreError: 436327:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [14097.380886] LustreError: 436323:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14097.414254] LustreError: 436323:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14102.507444] LustreError: 436322:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14102.548196] LustreError: 436322:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14107.632770] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14107.641432] LustreError: 436328:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14107.656192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14107.679603] Lustre: Skipped 1 previous similar message [14112.739239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14112.739279] LustreError: 438986:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14117.864625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14118.369210] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14118.595707] Lustre: server umount lustre-OST0000 complete [14122.980126] LustreError: 438986:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14123.026745] LustreError: 438986:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14130.052571] Lustre: server umount lustre-MDT0000 complete [14144.842947] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [14147.640741] Key type lgssc unregistered [14147.949091] LNet: 441082:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14147.961727] LNetError: 441082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14147.983648] LNet: Removed LNI 192.168.202.154@tcp [14149.165764] Key type .llcrypt unregistered [14149.168671] Key type ._llcrypt unregistered [14192.472963] Key type ._llcrypt registered [14192.478428] Key type .llcrypt registered [14192.666405] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [14193.056995] Key type .llcrypt unregistered [14193.058393] Key type ._llcrypt unregistered [14218.811594] Key type ._llcrypt registered [14218.816654] Key type .llcrypt registered [14219.167656] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [14221.333269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14221.549048] alg: No test for adler32 (adler32-zlib) [14222.679387] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [14223.167770] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [14224.976702] Key type lgssc registered [14226.834837] Lustre: Echo OBD driver; http://www.lustre.org/ [14256.831631] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14256.869380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14258.308469] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14258.367632] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14258.503604] Lustre: lustre-MDT0000: new disk, initializing [14258.625454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14258.660902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14263.562106] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [14274.816589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14274.966945] Lustre: 444464:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14275.019213] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14275.024819] Lustre: Skipped 1 previous similar message [14275.103107] Lustre: lustre-MDT0001: new disk, initializing [14275.232406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14275.266243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14275.293451] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14280.533454] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [14291.186448] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14298.047740] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14305.710358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14306.163531] Lustre: lustre-OST0000: new disk, initializing [14306.170754] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14306.176934] Lustre: 446431:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14306.313550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14312.664177] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [14313.078492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14313.107345] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14313.232384] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14322.009501] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14331.877212] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14331.883511] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14331.901730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14333.923826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14333.934908] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14336.863650] Lustre: server umount lustre-MDT0000 complete [14342.159612] LustreError: 444456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784058090 with bad export cookie 380071066575453879 [14342.172122] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14342.173526] LustreError: 444456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14342.644530] Lustre: server umount lustre-MDT0001 complete [14358.228755] Lustre: server umount lustre-OST0000 complete [14370.121276] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 15:41:56 (1784058116) [14378.016849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14378.399967] LustreError: 448459:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14378.421870] LustreError: 448459:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [14378.486700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14383.586557] LustreError: 448460:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14384.236875] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [14387.683850] LustreError: 448459:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14392.815753] LustreError: 448460:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14394.163448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14394.598296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14400.090222] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [14407.968488] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14415.088290] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14436.989963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14437.229600] Lustre: Found index 0 for lustre-OST0000, updating log [14437.493627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14444.632932] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [14449.796920] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14452.295196] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14460.386208] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14460.403570] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14460.407203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14460.422093] Lustre: Skipped 2 previous similar messages [14460.441180] Lustre: Skipped 3 previous similar messages [14465.366145] Lustre: server umount lustre-OST0000 complete [14487.018304] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14487.035462] LustreError: Skipped 1 previous similar message [14487.040704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14487.064123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14487.078659] Lustre: Skipped 2 previous similar messages [14489.457374] Lustre: server umount lustre-MDT0000 complete [14490.084440] LustreError: 449189:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14490.110856] LustreError: 449189:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14494.723318] LustreError: 448440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784058243 with bad export cookie 380071066575455272 [14494.735843] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14494.739958] LustreError: 448440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14495.271914] Lustre: server umount lustre-MDT0001 complete [14499.750223] LNet: 1 local NIs in recovery (showing 1): 192.168.202.154@tcp [14505.645708] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 15:44:11 (1784058251) [15136.286374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15136.829901] LustreError: 454874:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15136.945753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15141.969245] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [15142.386433] LustreError: 454875:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15146.478582] LustreError: 454874:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15151.173639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15151.604557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15156.634634] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [15164.257745] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15169.654705] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15189.869261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15190.240130] Lustre: Found index 0 for lustre-OST0000, updating log [15190.653317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15190.944149] LNet: 1 local NIs in recovery (showing 1): 192.168.202.154@tcp [15198.024709] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [15206.786581] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15209.134933] Lustre: server umount lustre-OST0000 complete [15214.499102] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15214.508099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15214.522790] Lustre: Skipped 2 previous similar messages [15214.540025] Lustre: Skipped 5 previous similar messages [15215.589548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15215.603767] Lustre: Skipped 1 previous similar message [15216.610834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15216.626382] Lustre: Skipped 1 previous similar message [15218.661385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15218.667604] Lustre: Skipped 3 previous similar messages [15219.688385] LustreError: 456690:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15219.707016] LustreError: 456690:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15219.862878] Lustre: server umount lustre-MDT0000 complete [15225.078284] LustreError: 454855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784058973 with bad export cookie 380071066575456357 [15225.087539] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15225.093781] LustreError: 454855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15225.900701] Lustre: server umount lustre-MDT0001 complete [15251.360439] LNet: 1 local NIs in recovery (showing 1): 192.168.202.154@tcp [15259.406765] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [15271.079686] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [15310.604531] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 15:57:36 (1784059056) [15330.486427] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [15333.712257] Key type lgssc unregistered [15334.088437] LNet: 461489:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15334.099073] LNetError: 461489:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15451.888138] LNet: Removed LNI 192.168.202.154@tcp [15452.573890] Key type .llcrypt unregistered [15452.575270] Key type ._llcrypt unregistered [15471.521791] Key type ._llcrypt registered [15471.524786] Key type .llcrypt registered [15471.691491] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [15473.240175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15473.411835] alg: No test for adler32 (adler32-zlib) [15474.625117] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [15474.931105] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [15476.624184] Key type lgssc registered [15477.937238] Lustre: Echo OBD driver; http://www.lustre.org/ [15483.288952] LNet: Added LNI 192.168.202.154@tcp100 [8/256/0/180] [15484.365711] LNet: Added LNI 192.168.202.154@tcp101 [8/256/0/180] [15485.616159] LNet: Added LNI 192.168.202.154@tcp102 [8/256/0/180] [15487.039611] LNet: Added LNI 192.168.202.154@tcp103 [8/256/0/180] [15488.627522] LNet: Added LNI 192.168.202.154@tcp104 [8/256/0/180] [15490.250968] LNet: Added LNI 192.168.202.154@tcp105 [8/256/0/180] [15491.438618] LNet: Added LNI 192.168.202.154@tcp106 [8/256/0/180] [15492.772141] LNet: Added LNI 192.168.202.154@tcp107 [8/256/0/180] [15494.291616] LNet: Added LNI 192.168.202.154@tcp108 [8/256/0/180] [15495.718122] LNet: Added LNI 192.168.202.154@tcp109 [8/256/0/180] [15496.775101] LNet: Added LNI 192.168.202.154@tcp110 [8/256/0/180] [15497.643581] LNet: Added LNI 192.168.202.154@tcp111 [8/256/0/180] [15498.448989] LNet: Added LNI 192.168.202.154@tcp112 [8/256/0/180] [15499.424498] LNet: Added LNI 192.168.202.154@tcp113 [8/256/0/180] [15500.751138] LNet: Added LNI 192.168.202.154@tcp114 [8/256/0/180] [15502.026648] LNet: Added LNI 192.168.202.154@tcp115 [8/256/0/180] [15503.112535] LNet: Added LNI 192.168.202.154@tcp116 [8/256/0/180] [15504.204553] LNet: Added LNI 192.168.202.154@tcp117 [8/256/0/180] [15505.091823] LNet: Added LNI 192.168.202.154@tcp118 [8/256/0/180] [15505.964439] LNet: Added LNI 192.168.202.154@tcp119 [8/256/0/180] [15506.927951] LNet: Added LNI 192.168.202.154@tcp120 [8/256/0/180] [15507.780545] LNet: Added LNI 192.168.202.154@tcp121 [8/256/0/180] [15508.884062] LNet: Added LNI 192.168.202.154@tcp122 [8/256/0/180] [15510.011445] LNet: Added LNI 192.168.202.154@tcp123 [8/256/0/180] [15511.048268] LNet: Added LNI 192.168.202.154@tcp124 [8/256/0/180] [15512.198709] LNet: Added LNI 192.168.202.154@tcp125 [8/256/0/180] [15513.569410] LNet: Added LNI 192.168.202.154@tcp126 [8/256/0/180] [15515.276927] LNet: Added LNI 192.168.202.154@tcp127 [8/256/0/180] [15516.268875] LNet: Added LNI 192.168.202.154@tcp128 [8/256/0/180] [15517.503370] LNet: Added LNI 192.168.202.154@tcp129 [8/256/0/180] [15518.664251] LNet: Added LNI 192.168.202.154@tcp130 [8/256/0/180] [15519.879659] LNet: Added LNI 192.168.202.154@tcp131 [8/256/0/180] [15520.955187] LNet: Added LNI 192.168.202.154@tcp132 [8/256/0/180] [15521.857756] LNet: Added LNI 192.168.202.154@tcp133 [8/256/0/180] [15522.966327] LNet: Added LNI 192.168.202.154@tcp134 [8/256/0/180] [15523.968619] LNet: Added LNI 192.168.202.154@tcp135 [8/256/0/180] [15530.490904] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15530.552781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15531.721058] Lustre: 464534:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.154@tcp: can't add NID 192.168.202.154@tcp130: rc = -75 [15531.819218] LNetError: 464556:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.154 [15531.835794] LNetError: Protocol error connecting to 192.168.202.154@tcp121 at host 192.168.202.154:988: is it running a compatible version of Lustre? [15531.844120] LNetError: Skipped 1 previous similar message [15531.979222] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15532.021516] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15532.100305] Lustre: lustre-MDT0000: new disk, initializing [15532.221917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15532.237919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15532.386159] LNet: 9 local NIs in recovery (showing 9): 192.168.202.154@tcp118, 192.168.202.154@tcp121, 192.168.202.154@tcp123, 192.168.202.154@tcp125, 192.168.202.154@tcp126, 192.168.202.154@tcp127, 192.168.202.154@tcp128, 192.168.202.154@tcp122, 192.168.202.154@tcp129 [15532.390911] LNetError: 464554:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.154 [15532.421427] LNetError: 464554:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 13 previous similar messages [15532.441361] LNetError: Protocol error connecting to 192.168.202.154@tcp115 at host 192.168.202.154:988: is it running a compatible version of Lustre? [15532.463839] LNetError: Skipped 18 previous similar messages [15536.233958] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [15547.673454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15547.883906] Lustre: 464552:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15547.996418] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15548.003759] Lustre: Skipped 1 previous similar message [15548.142916] Lustre: lustre-MDT0001: new disk, initializing [15548.188897] Lustre: 465354:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.154@tcp: can't add NID 192.168.202.154@tcp130: rc = -75 [15548.207077] Lustre: 465354:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15548.385894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15548.431422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15548.443094] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15553.686118] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [15557.747235] Lustre: 465778:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.154@tcp: can't add NID 192.168.202.154@tcp130: rc = -75 [15557.762740] Lustre: 465778:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15561.960544] LNetError: 461981:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.154 [15561.970890] LNetError: 461981:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 6 previous similar messages [15561.997579] LNetError: Protocol error connecting to 192.168.202.154@tcp119 at host 192.168.202.154:988: is it running a compatible version of Lustre? [15562.917601] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15564.002885] LNetError: 461980:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.154 [15564.011269] LNetError: 461980:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [15564.017733] LNetError: Protocol error connecting to 192.168.202.154@tcp122 at host 192.168.202.154:988: is it running a compatible version of Lustre? [15564.025652] LNetError: Skipped 1 previous similar message [15568.950857] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15569.121766] LNetError: 464563:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.154 [15569.142080] LNetError: 464563:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [15569.164421] LNetError: Protocol error connecting to 192.168.202.154@tcp118 at host 192.168.202.154:988: is it running a compatible version of Lustre? [15569.174725] LNetError: Skipped 2 previous similar messages [15577.573553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15578.199604] Lustre: lustre-OST0000: new disk, initializing [15578.207393] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15578.217237] Lustre: 466527:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15578.341139] LNetError: 464556:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.154 [15578.356897] LNetError: 464556:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 8 previous similar messages [15578.375585] Lustre: 466532:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.154@tcp: can't add NID 192.168.202.154@tcp130: rc = -75 [15578.376550] LNetError: Protocol error connecting to 192.168.202.154@tcp120 at host 192.168.202.154:988: is it running a compatible version of Lustre? [15578.390525] Lustre: 466532:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15578.428501] LNetError: Skipped 8 previous similar messages [15578.472141] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15585.317482] Lustre: 466821:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.154@tcp: can't add NID 192.168.202.154@tcp130: rc = -75 [15585.331843] Lustre: 466821:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15585.394385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15585.403275] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15585.602399] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15586.617457] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [15594.728327] LNetError: 464555:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.154 [15594.744253] LNetError: 464555:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 11 previous similar messages [15594.752822] LNetError: Protocol error connecting to 192.168.202.154@tcp118 at host 192.168.202.154:988: is it running a compatible version of Lustre? [15594.771614] LNetError: Skipped 11 previous similar messages [15599.492629] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15609.318282] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15609.322974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15609.327038] Lustre: Skipped 1 previous similar message [15614.440923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15614.455799] Lustre: Skipped 2 previous similar messages [15615.016693] Lustre: server umount lustre-OST0000 complete [15624.681750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15624.692500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15624.700944] Lustre: Skipped 1 previous similar message [15624.724491] Lustre: Skipped 1 previous similar message [15627.400710] Lustre: server umount lustre-MDT0000 complete [15627.568581] LNetError: 461982:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.154 [15627.583812] LNetError: 461982:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 16 previous similar messages [15627.598700] LNetError: Protocol error connecting to 192.168.202.154@tcp122 at host 192.168.202.154:988: is it running a compatible version of Lustre? [15627.612597] LNetError: Skipped 16 previous similar messages [15632.045895] LustreError: 464542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784059380 with bad export cookie 16636131596398359745 [15632.047809] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15632.078795] LustreError: 464542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15632.571555] Lustre: server umount lustre-MDT0001 complete [15644.824469] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [15647.322875] Key type lgssc unregistered [15647.766932] LNet: 468442:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15647.773178] LNetError: 468442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15648.800189] LNet: Removed LNI 192.168.202.154@tcp [15648.807594] LNetError: 468442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15648.828162] LNet: Removed LNI 192.168.202.154@tcp100 [15649.888566] LNet: Removed LNI 192.168.202.154@tcp101 [15649.902899] LNetError: 468442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15649.907123] LNetError: 468442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15649.916208] LNet: Removed LNI 192.168.202.154@tcp102 [15650.977061] LNet: Removed LNI 192.168.202.154@tcp103 [15652.000312] LNet: Removed LNI 192.168.202.154@tcp104 [15652.003462] LNetError: 468442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15652.028371] LNetError: 468442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15652.043460] LNet: Removed LNI 192.168.202.154@tcp105 [15653.088741] LNet: Removed LNI 192.168.202.154@tcp106 [15654.112474] LNet: Removed LNI 192.168.202.154@tcp107 [15655.136168] LNet: Removed LNI 192.168.202.154@tcp108 [15656.160523] LNet: Removed LNI 192.168.202.154@tcp109 [15656.169734] LNetError: 468442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15656.185358] LNetError: 468442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15656.218697] LNet: Removed LNI 192.168.202.154@tcp110 [15657.248389] LNet: Removed LNI 192.168.202.154@tcp111 [15658.273378] LNet: Removed LNI 192.168.202.154@tcp112 [15659.296167] LNet: Removed LNI 192.168.202.154@tcp113 [15660.320429] LNet: Removed LNI 192.168.202.154@tcp114 [15661.344163] LNet: Removed LNI 192.168.202.154@tcp115 [15662.368197] LNet: Removed LNI 192.168.202.154@tcp116 [15663.392176] LNet: Removed LNI 192.168.202.154@tcp117 [15664.416307] LNet: Removed LNI 192.168.202.154@tcp118 [15664.418944] LNetError: 468442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15664.424618] LNetError: 468442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15664.442735] LNet: Removed LNI 192.168.202.154@tcp119 [15665.506843] LNet: Removed LNI 192.168.202.154@tcp120 [15666.528206] LNet: Removed LNI 192.168.202.154@tcp121 [15667.552270] LNet: Removed LNI 192.168.202.154@tcp122 [15668.576211] LNet: Removed LNI 192.168.202.154@tcp123 [15669.600153] LNet: Removed LNI 192.168.202.154@tcp124 [15670.624238] LNet: Removed LNI 192.168.202.154@tcp125 [15671.648177] LNet: Removed LNI 192.168.202.154@tcp126 [15672.674165] LNet: Removed LNI 192.168.202.154@tcp127 [15673.696365] LNet: Removed LNI 192.168.202.154@tcp128 [15674.721534] LNet: Removed LNI 192.168.202.154@tcp129 [15674.731645] LNet: Removed LNI 192.168.202.154@tcp130 [15674.742697] LNet: Removed LNI 192.168.202.154@tcp131 [15674.744507] LNet: Removed LNI 192.168.202.154@tcp132 [15674.753674] LNet: Removed LNI 192.168.202.154@tcp133 [15674.761446] LNet: Removed LNI 192.168.202.154@tcp134 [15674.793661] LNet: Removed LNI 192.168.202.154@tcp135 [15675.995712] Key type .llcrypt unregistered [15675.996857] Key type ._llcrypt unregistered [15706.369356] Key type ._llcrypt registered [15706.371949] Key type .llcrypt registered [15706.494153] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [15724.441662] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [15726.272113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15726.409943] alg: No test for adler32 (adler32-zlib) [15727.717609] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [15728.082058] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [15729.712282] Key type lgssc registered [15731.363488] Lustre: Echo OBD driver; http://www.lustre.org/ [15765.066516] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 16:05:11 (1784059511) [15784.012512] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [15786.411818] Key type lgssc unregistered [15786.649948] LNet: 472534:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15786.657307] LNetError: 472534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15786.670688] LNet: Removed LNI 192.168.202.154@tcp [15787.474757] Key type .llcrypt unregistered [15787.476654] Key type ._llcrypt unregistered [15807.628917] Key type ._llcrypt registered [15807.631187] Key type .llcrypt registered [15807.829247] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [15809.452946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15809.783910] alg: No test for adler32 (adler32-zlib) [15810.980862] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [15811.259690] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [15812.904436] Key type lgssc registered [15815.215543] Lustre: Echo OBD driver; http://www.lustre.org/ [15826.441256] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15826.491716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15827.781766] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15827.820342] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15827.947327] Lustre: lustre-MDT0000: new disk, initializing [15828.029931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15828.046653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15832.896418] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [15843.608256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15843.707370] Lustre: 473749:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15843.785479] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15843.789966] Lustre: Skipped 1 previous similar message [15843.906103] Lustre: lustre-MDT0001: new disk, initializing [15844.065476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15844.113517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15844.117357] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15849.079353] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [15861.288601] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15867.637320] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15874.838830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15875.197697] Lustre: lustre-OST0000: new disk, initializing [15875.200863] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15875.207121] Lustre: 475718:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15875.274679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15881.595714] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [15884.883895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15884.898642] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15885.032588] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15893.921710] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15898.348807] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing lnet_if_list [15904.053620] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15904.080149] LNet: Added LNI 192.168.202.154@tcp10 [8/256/0/180] [15911.165967] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15911.202688] LNet: Added LNI 192.168.202.154@tcp11 [8/256/0/180] [15916.215130] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15916.247128] LNet: Added LNI 192.168.202.154@tcp12 [8/256/0/180] [15920.883561] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15920.924730] LNet: Added LNI 192.168.202.154@tcp13 [8/256/0/180] [15925.202445] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15925.218727] LNet: Added LNI 192.168.202.154@tcp14 [8/256/0/180] [15930.989715] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15931.049997] LNet: Added LNI 192.168.202.154@tcp15 [8/256/0/180] [15938.240832] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15938.326398] LNet: Added LNI 192.168.202.154@tcp16 [8/256/0/180] [15945.592337] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15945.665921] LNet: Added LNI 192.168.202.154@tcp17 [8/256/0/180] [15951.205281] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15951.256298] LNet: Added LNI 192.168.202.154@tcp18 [8/256/0/180] [15957.298605] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15957.338300] LNet: Added LNI 192.168.202.154@tcp19 [8/256/0/180] [15964.509533] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15964.565272] LNet: Added LNI 192.168.202.154@tcp20 [8/256/0/180] [15973.810453] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15973.882727] LNet: Added LNI 192.168.202.154@tcp21 [8/256/0/180] [15981.207737] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15981.233312] LNet: Added LNI 192.168.202.154@tcp22 [8/256/0/180] [15989.307842] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15989.359613] LNet: Added LNI 192.168.202.154@tcp23 [8/256/0/180] [15998.032522] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15998.065428] LNet: Added LNI 192.168.202.154@tcp24 [8/256/0/180] [16007.023675] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [16007.094347] LNet: Added LNI 192.168.202.154@tcp25 [8/256/0/180] [16014.409143] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [16014.430599] LNet: Added LNI 192.168.202.154@tcp26 [8/256/0/180] [16021.549124] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [16021.570860] LNet: Added LNI 192.168.202.154@tcp27 [8/256/0/180] [16028.840727] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [16028.869465] LNet: Added LNI 192.168.202.154@tcp28 [8/256/0/180] [16035.871750] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [16035.911976] LNet: Added LNI 192.168.202.154@tcp29 [8/256/0/180] [16042.351142] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [16042.392525] LNet: Added LNI 192.168.202.154@tcp30 [8/256/0/180] [16049.083939] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [16049.117960] LNet: Added LNI 192.168.202.154@tcp31 [8/256/0/180] [16056.104883] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [16056.155714] LNet: Added LNI 192.168.202.154@tcp32 [8/256/0/180] [16062.380906] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [16062.407509] LNet: Added LNI 192.168.202.154@tcp33 [8/256/0/180] [16068.571800] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [16068.618978] LNet: Added LNI 192.168.202.154@tcp34 [8/256/0/180] [16075.394905] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [16075.426454] LNet: Added LNI 192.168.202.154@tcp35 [8/256/0/180] [16083.203392] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [16083.230100] LNet: Added LNI 192.168.202.154@tcp36 [8/256/0/180] [16092.619552] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [16092.694401] LNet: Added LNI 192.168.202.154@tcp37 [8/256/0/180] [16102.191176] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [16102.260593] LNet: Added LNI 192.168.202.154@tcp38 [8/256/0/180] [16112.400419] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [16112.455113] LNet: Added LNI 192.168.202.154@tcp39 [8/256/0/180] [16122.312994] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [16122.366360] LNet: Added LNI 192.168.202.154@tcp40 [8/256/0/180] [16132.741818] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [16132.780969] LNet: Added LNI 192.168.202.154@tcp41 [8/256/0/180] [16140.950861] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [16140.974667] LNet: Added LNI 192.168.202.154@tcp42 [8/256/0/180] [16149.900756] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [16149.929824] LNet: Added LNI 192.168.202.154@tcp43 [8/256/0/180] [16158.095953] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [16158.142128] LNet: Added LNI 192.168.202.154@tcp44 [8/256/0/180] [16167.053209] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [16167.084175] LNet: Added LNI 192.168.202.154@tcp45 [8/256/0/180] [16174.834710] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [16174.873663] LNet: Added LNI 192.168.202.154@tcp46 [8/256/0/180] [16183.799388] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [16183.831465] LNet: Added LNI 192.168.202.154@tcp47 [8/256/0/180] [16192.339993] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [16192.379789] LNet: Added LNI 192.168.202.154@tcp48 [8/256/0/180] [16200.921557] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [16200.967130] LNet: Added LNI 192.168.202.154@tcp49 [8/256/0/180] [16210.805365] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [16210.842505] LNet: Added LNI 192.168.202.154@tcp50 [8/256/0/180] [16233.444837] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16233.455046] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16233.464858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16233.484963] Lustre: Skipped 2 previous similar messages [16237.236290] Lustre: server umount lustre-MDT0000 complete [16241.280065] LustreError: 473742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784059989 with bad export cookie 7362154502817904768 [16241.295534] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16241.310116] LustreError: 473742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16241.706325] Lustre: server umount lustre-MDT0001 complete [16260.066257] Lustre: 473019:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784059992/real 1784059992] req@0000000093c64ef8 x1870722036322176/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784060008 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16260.095468] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16260.112779] Lustre: Skipped 2 previous similar messages [16261.261907] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [16263.136409] Lustre: 473021:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784059995/real 1784059995] req@000000005adc04fe x1870722036322304/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784060011 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16265.192690] Lustre: 473021:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784059997/real 1784059997] req@00000000e5852fa1 x1870722036322432/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784060013 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16267.232303] Lustre: 473020:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784059999/real 1784059999] req@0000000083f051ea x1870722036322688/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784060015 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16270.313545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16272.353311] Lustre: 473019:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784060004/real 1784060004] req@00000000e9f605d5 x1870722036322816/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784060020 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16286.094176] LustreError: 475711:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16286.113316] LustreError: 475711:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [16286.196818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16289.186783] LustreError: 475712:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16290.212193] LustreError: 475711:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16290.213349] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16291.416065] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [16295.403719] LustreError: 475712:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16300.446603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16300.519655] LustreError: 475711:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16300.935453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16305.014278] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16305.066525] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [16313.488420] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16321.041727] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16340.964350] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16340.965499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16340.977936] Lustre: Skipped 1 previous similar message [16340.985390] Lustre: Skipped 1 previous similar message [16346.096600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16346.643470] Lustre: server umount lustre-OST0000 complete [16351.202914] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16351.218366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16351.240867] Lustre: Skipped 1 previous similar message [16351.275558] Lustre: Skipped 2 previous similar messages [16356.322196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16356.328031] Lustre: Skipped 1 previous similar message [16357.171363] Lustre: server umount lustre-MDT0000 complete [16361.424316] LustreError: 473743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784060109 with bad export cookie 7362154502817907505 [16361.431785] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16361.441182] LustreError: 473743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16361.750325] Lustre: server umount lustre-MDT0001 complete [16372.691776] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [16375.561157] Key type lgssc unregistered [16376.056427] LNet: 502749:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16376.077043] LNetError: 502749:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16376.084892] LNet: Removed LNI 192.168.202.154@tcp [16376.090291] LNet: Removed LNI 192.168.202.154@tcp10 [16376.113701] LNet: Removed LNI 192.168.202.154@tcp11 [16376.120782] LNet: Removed LNI 192.168.202.154@tcp12 [16376.136641] LNet: Removed LNI 192.168.202.154@tcp13 [16376.151345] LNet: Removed LNI 192.168.202.154@tcp14 [16376.153567] LNet: Removed LNI 192.168.202.154@tcp15 [16376.170121] LNet: Removed LNI 192.168.202.154@tcp16 [16376.181635] LNet: Removed LNI 192.168.202.154@tcp17 [16376.196574] LNet: Removed LNI 192.168.202.154@tcp18 [16376.200158] LNet: Removed LNI 192.168.202.154@tcp19 [16376.202468] LNet: Removed LNI 192.168.202.154@tcp20 [16376.205772] LNet: Removed LNI 192.168.202.154@tcp21 [16376.231964] LNet: Removed LNI 192.168.202.154@tcp22 [16376.237338] LNet: Removed LNI 192.168.202.154@tcp23 [16376.240713] LNet: Removed LNI 192.168.202.154@tcp24 [16376.242943] LNet: Removed LNI 192.168.202.154@tcp25 [16376.247940] LNet: Removed LNI 192.168.202.154@tcp26 [16376.254399] LNet: Removed LNI 192.168.202.154@tcp27 [16376.257499] LNet: Removed LNI 192.168.202.154@tcp28 [16376.264917] LNet: Removed LNI 192.168.202.154@tcp29 [16376.270789] LNet: Removed LNI 192.168.202.154@tcp30 [16376.282485] LNet: Removed LNI 192.168.202.154@tcp31 [16376.286432] LNet: Removed LNI 192.168.202.154@tcp32 [16376.299639] LNet: Removed LNI 192.168.202.154@tcp33 [16376.301764] LNet: Removed LNI 192.168.202.154@tcp34 [16376.310747] LNet: Removed LNI 192.168.202.154@tcp35 [16376.317686] LNet: Removed LNI 192.168.202.154@tcp36 [16376.321873] LNet: Removed LNI 192.168.202.154@tcp37 [16376.325944] LNet: Removed LNI 192.168.202.154@tcp38 [16376.329864] LNet: Removed LNI 192.168.202.154@tcp39 [16376.337585] LNet: Removed LNI 192.168.202.154@tcp40 [16376.342127] LNet: Removed LNI 192.168.202.154@tcp41 [16376.343870] LNet: Removed LNI 192.168.202.154@tcp42 [16376.348770] LNet: Removed LNI 192.168.202.154@tcp43 [16376.356849] LNet: Removed LNI 192.168.202.154@tcp44 [16376.363535] LNet: Removed LNI 192.168.202.154@tcp45 [16376.371717] LNet: Removed LNI 192.168.202.154@tcp46 [16376.378609] LNet: Removed LNI 192.168.202.154@tcp47 [16376.380759] LNet: Removed LNI 192.168.202.154@tcp48 [16376.389928] LNet: Removed LNI 192.168.202.154@tcp49 [16376.406078] LNet: Removed LNI 192.168.202.154@tcp50 [16377.290971] Key type .llcrypt unregistered [16377.293672] Key type ._llcrypt unregistered [16403.069951] Key type ._llcrypt registered [16403.073735] Key type .llcrypt registered [16403.216753] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [16419.985470] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [16421.155339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16421.179896] alg: No test for adler32 (adler32-zlib) [16422.325148] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [16422.616898] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [16424.360153] Key type lgssc registered [16426.351555] Lustre: Echo OBD driver; http://www.lustre.org/ [16466.542976] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16466.593672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16467.981895] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16468.004617] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16468.066512] Lustre: lustre-MDT0000: new disk, initializing [16468.118300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16468.136418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16473.080969] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [16484.238177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16484.458409] Lustre: 505948:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16484.591648] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16484.601244] Lustre: Skipped 1 previous similar message [16484.689545] Lustre: lustre-MDT0001: new disk, initializing [16484.809256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16484.891428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16484.930420] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16489.736678] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [16500.400685] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16506.935097] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16516.282755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16516.467391] Lustre: *** cfs_fail_loc=90f, val=0*** [16516.537641] Lustre: lustre-OST0000: new disk, initializing [16516.544347] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16516.550598] Lustre: 507963:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16516.652313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16522.924704] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [16524.392538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16524.416745] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16524.566763] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16533.288401] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16545.255447] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16545.256144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16545.269065] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16545.274462] Lustre: Skipped 1 previous similar message [16545.306765] Lustre: Skipped 1 previous similar message [16548.644556] Lustre: server umount lustre-OST0000 complete [16555.495153] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16555.510238] Lustre: Skipped 1 previous similar message [16555.522804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16555.534323] Lustre: Skipped 1 previous similar message [16561.970744] Lustre: server umount lustre-MDT0000 complete [16565.837316] LustreError: 506741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784060314 with bad export cookie 8653612858004875789 [16565.844220] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16565.853661] LustreError: 506741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16566.428557] Lustre: server umount lustre-MDT0001 complete [16578.788995] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [16582.236708] Key type lgssc unregistered [16582.595829] LNet: 509872:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16582.600306] LNetError: 509872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16582.624934] LNet: Removed LNI 192.168.202.154@tcp [16583.770445] Key type .llcrypt unregistered [16583.776646] Key type ._llcrypt unregistered [16606.652314] Key type ._llcrypt registered [16606.656845] Key type .llcrypt registered [16606.765859] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [16608.251450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16608.302720] alg: No test for adler32 (adler32-zlib) [16609.338736] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [16609.536508] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [16611.184289] Key type lgssc registered [16612.493680] Lustre: Echo OBD driver; http://www.lustre.org/ [16621.028386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16621.074675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16622.650871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16627.837489] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [16632.804757] LustreError: 511144:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16632.834514] LustreError: 511144:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16637.606929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16637.927358] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [16638.215799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16643.391525] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [16651.436758] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16658.061364] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16666.432720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16667.042331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16674.132731] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [16682.330751] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16693.386232] loop0: detected capacity change from 0 to 204800000 [16693.630803] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16693.645758] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16693.673435] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16700.847745] loop0: detected capacity change from 0 to 204800000 [16701.129740] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16701.141756] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16701.171248] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16708.265067] loop0: detected capacity change from 0 to 204800000 [16713.701020] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16713.835480] Lustre: fs15246-OST0000: new disk, initializing [16713.842766] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16713.849341] Lustre: 514736:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16713.899274] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16721.199158] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [16731.345459] loop1: detected capacity change from 0 to 204800000 [16737.140555] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16737.284084] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16737.326838] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16737.387914] Lustre: fs15246-MDT0000: new disk, initializing [16737.604345] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16737.638906] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16737.816891] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16738.852038] Lustre: 515859:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000009a1b3c10 x1870722873999488/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1784060503 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16742.618736] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [16747.942901] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16747.984208] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16748.037921] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16765.427573] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16765.464059] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16770.540589] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16771.212494] Lustre: server umount fs15246-MDT0000 complete [16775.654504] LustreError: 512305:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16780.772130] LustreError: 511140:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16781.053713] Lustre: server umount fs15246-OST0000 complete [16796.133941] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16796.136759] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16796.168592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16801.251092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16801.257979] Lustre: Skipped 1 previous similar message [16806.881044] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16807.160980] Lustre: server umount lustre-OST0000 complete [16811.490749] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16811.499401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16811.509723] Lustre: Skipped 2 previous similar messages [16811.531472] Lustre: Skipped 3 previous similar messages [16817.771619] Lustre: server umount lustre-MDT0000 complete [16821.741464] LustreError: 511144:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16821.758229] LustreError: 511144:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16822.545195] LustreError: 511123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784060571 with bad export cookie 9370895678151112362 [16822.559851] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16822.572076] LustreError: 511123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16823.681777] Lustre: server umount lustre-MDT0001 complete [16839.457876] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [16842.982124] Key type lgssc unregistered [16843.392405] LNet: 518959:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16843.400872] LNetError: 518959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16844.456342] LNet: Removed LNI 192.168.202.154@tcp [16845.675962] Key type .llcrypt unregistered [16845.678138] Key type ._llcrypt unregistered [16868.073441] Key type ._llcrypt registered [16868.076951] Key type .llcrypt registered [16868.250717] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [16870.120449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16870.373709] alg: No test for adler32 (adler32-zlib) [16871.955538] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [16872.406556] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [16874.152236] Key type lgssc registered [16876.185805] Lustre: Echo OBD driver; http://www.lustre.org/ [16923.729878] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [16934.337341] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [16973.855574] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16973.878951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16975.175544] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16975.195601] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16975.266401] Lustre: lustre-MDT0000: new disk, initializing [16975.342091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16975.358941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16979.750949] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [16991.729093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16991.815248] Lustre: 523434:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16991.878729] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16991.882574] Lustre: Skipped 1 previous similar message [16991.981650] Lustre: lustre-MDT0001: new disk, initializing [16992.131625] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16992.173810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16992.186028] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16997.113087] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [17006.286627] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17012.292871] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17018.666252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17018.962354] Lustre: lustre-OST0000: new disk, initializing [17018.971589] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [17018.982320] Lustre: 525406:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17019.101195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17024.579151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [17024.594799] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [17024.719181] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [17025.770331] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [17035.691281] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17041.846651] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [17042.086953] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [17046.882482] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [17047.102174] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [17050.594932] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17050.600870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17050.601301] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [17050.611521] Lustre: Skipped 1 previous similar message [17050.644398] Lustre: Skipped 1 previous similar message [17055.591347] Lustre: server umount lustre-OST0000 complete [17060.834129] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17060.862987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17060.867117] Lustre: Skipped 1 previous similar message [17065.701102] Lustre: server umount lustre-MDT0000 complete [17069.905833] LustreError: 523427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784060818 with bad export cookie 3130437187825936192 [17069.908120] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17069.924027] LustreError: 523427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [17070.486641] Lustre: server umount lustre-MDT0001 complete [17094.866599] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [17105.424638] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [17139.037238] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16799 sec ========= 16:28:04 (1784060884) [17141.164756] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 16:28:06 (1784060886) === [17145.099265] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 16:28:10 (1784060890) === [17187.200284] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [17190.311505] Key type lgssc unregistered [17190.658191] LNet: 532049:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17190.665242] LNetError: 532049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17190.682336] LNet: Removed LNI 192.168.202.154@tcp [17191.637495] Key type .llcrypt unregistered [17191.640118] Key type ._llcrypt unregistered