[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 472117660 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003261] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010141] LSM: Security Framework initializing [ 0.011058] Yama: becoming mindful. [ 0.012042] SELinux: Initializing. [ 0.014036] *** VALIDATE selinux *** [ 0.022821] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027546] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029114] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030116] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031123] *** VALIDATE tmpfs *** [ 0.033753] *** VALIDATE proc *** [ 0.034259] *** VALIDATE cgroup *** [ 0.035013] *** VALIDATE cgroup2 *** [ 0.036294] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037163] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039030] Spectre V2 : User space: Vulnerable [ 0.040007] Speculative Store Bypass: Vulnerable [ 0.043000] debug: unmapping init [mem 0xffffffff85059000-0xffffffff85060fff] [ 0.043194] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044999] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046025] ... version: 2 [ 0.047009] ... bit width: 48 [ 0.048013] ... generic registers: 4 [ 0.049016] ... value mask: 0000ffffffffffff [ 0.050014] ... max period: 00007fffffffffff [ 0.051015] ... fixed-purpose events: 3 [ 0.052011] ... event mask: 000000070000000f [ 0.053700] rcu: Hierarchical SRCU implementation. [ 0.055705] smp: Bringing up secondary CPUs ... [ 0.056561] x86: Booting SMP configuration: [ 0.057020] .... node #0, CPUs: #1 #2 #3 [ 0.064412] smp: Brought up 1 node, 4 CPUs [ 0.066018] smpboot: Max logical packages: 1 [ 0.067016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.105000] node 0 deferred pages initialised in 28ms [ 0.111019] devtmpfs: initialized [ 0.112248] x86/mm: Memory block size: 128MB [ 0.115651] gcov: version magic: 0x41383552 [ 0.116538] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.119086] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.123738] pinctrl core: initialized pinctrl subsystem [ 0.126195] [ 0.127010] ************************************************************* [ 0.130014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.132011] ** ** [ 0.135018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.138015] ** ** [ 0.141015] ** This means that this kernel is built to expose internal ** [ 0.144013] ** IOMMU data structures, which may compromise security on ** [ 0.145013] ** your system. ** [ 0.147015] ** ** [ 0.150016] ** If you see this message and you are not debugging the ** [ 0.153018] ** kernel, report this immediately to your vendor! ** [ 0.156015] ** ** [ 0.159012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161011] ************************************************************* [ 0.164126] NET: Registered protocol family 16 [ 0.165746] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.168068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.172076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.174113] cpuidle: using governor menu [ 0.182651] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.185601] PCI: Using configuration type 1 for base access [ 0.188244] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.197666] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.201018] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.205046] cryptd: max_cpu_qlen set to 1000 [ 0.207226] ACPI: Added _OSI(Module Device) [ 0.208014] ACPI: Added _OSI(Processor Device) [ 0.210019] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.211208] ACPI: Added _OSI(Processor Aggregator Device) [ 0.215759] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.224603] ACPI: Interpreter enabled [ 0.227704] ACPI: PM: (supports S0 S3 S4 S5) [ 0.229010] ACPI: Using IOAPIC for interrupt routing [ 0.232358] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.237675] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.257585] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.271055] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.284024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.295086] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.313000] acpiphp: Slot [2] registered [ 0.315183] acpiphp: Slot [5] registered [ 0.321162] acpiphp: Slot [6] registered [ 0.324275] acpiphp: Slot [7] registered [ 0.327597] acpiphp: Slot [8] registered [ 0.330121] acpiphp: Slot [9] registered [ 0.331104] acpiphp: Slot [10] registered [ 0.333118] acpiphp: Slot [3] registered [ 0.338526] acpiphp: Slot [4] registered [ 0.342410] acpiphp: Slot [11] registered [ 0.348141] acpiphp: Slot [12] registered [ 0.351169] acpiphp: Slot [13] registered [ 0.355143] acpiphp: Slot [14] registered [ 0.358137] acpiphp: Slot [15] registered [ 0.360173] acpiphp: Slot [16] registered [ 0.363165] acpiphp: Slot [17] registered [ 0.365136] acpiphp: Slot [18] registered [ 0.375267] acpiphp: Slot [19] registered [ 0.378144] acpiphp: Slot [20] registered [ 0.384370] acpiphp: Slot [21] registered [ 0.387471] acpiphp: Slot [22] registered [ 0.391131] acpiphp: Slot [23] registered [ 0.393116] acpiphp: Slot [24] registered [ 0.396375] acpiphp: Slot [25] registered [ 0.401147] acpiphp: Slot [26] registered [ 0.405157] acpiphp: Slot [27] registered [ 0.410158] acpiphp: Slot [28] registered [ 0.416337] acpiphp: Slot [29] registered [ 0.422147] acpiphp: Slot [30] registered [ 0.425161] acpiphp: Slot [31] registered [ 0.428082] PCI host bridge to bus 0000:00 [ 0.432033] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.435027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.441028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.449031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.454034] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.458033] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.463443] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.467937] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.471291] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.486015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.494738] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.497016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.499015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.501015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.509906] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.517935] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.524019] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.525000] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.531014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.559018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.566016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.571765] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.581021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.588022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.606021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.627082] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.636018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.651022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.682020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.691452] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.712027] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.726018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.744110] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.778021] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.814017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.824018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.865034] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.901020] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.921018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.937017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.952021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.962576] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.979018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.990017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.012017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.029184] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.037590] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.041399] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.047284] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.056517] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.068095] iommu: Default domain type: Passthrough [ 1.073575] SCSI subsystem initialized [ 1.076212] ACPI: bus type USB registered [ 1.079118] usbcore: registered new interface driver usbfs [ 1.084141] usbcore: registered new interface driver hub [ 1.089102] usbcore: registered new device driver usb [ 1.097048] pps_core: LinuxPPS API ver. 1 registered [ 1.102014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.108065] PTP clock support registered [ 1.120106] EDAC MC: Ver: 3.0.0 [ 1.121223] PCI: Using ACPI for IRQ routing [ 1.122887] NetLabel: Initializing [ 1.126013] NetLabel: domain hash size = 128 [ 1.129013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.134091] NetLabel: unlabeled traffic allowed by default [ 1.142207] vgaarb: loaded [ 1.149414] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.153023] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.175583] clocksource: Switched to clocksource kvm-clock [ 1.514841] VFS: Disk quotas dquot_6.6.0 [ 1.519639] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.527331] *** VALIDATE ramfs *** [ 1.535043] *** VALIDATE hugetlbfs *** [ 1.537798] pnp: PnP ACPI init [ 1.542635] pnp: PnP ACPI: found 6 devices [ 1.561338] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.565356] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.568219] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.571417] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.576156] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.578580] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.583331] NET: Registered protocol family 2 [ 1.588891] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.596935] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.605619] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.622414] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.632832] TCP: Hash tables configured (established 65536 bind 65536) [ 1.642110] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.653664] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.662391] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.677795] NET: Registered protocol family 1 [ 1.694170] RPC: Registered named UNIX socket transport module. [ 1.705596] RPC: Registered udp transport module. [ 1.710021] RPC: Registered tcp transport module. [ 1.713760] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.718477] NET: Registered protocol family 44 [ 1.725978] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.734924] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.743373] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.750855] PCI: CLS 0 bytes, default 64 [ 1.754765] Unpacking initramfs... [ 5.881800] debug: unmapping init [mem 0xffff8a7c7cc54000-0xffff8a7c7ffbffff] [ 5.894749] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 5.903530] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 5.910347] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 6.327042] hrtimer: interrupt took 8273476 ns [ 7.858105] Initialise system trusted keyrings [ 7.866101] Key type blacklist registered [ 7.875108] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 7.887171] zbud: loaded [ 7.898056] *** VALIDATE nfs *** [ 7.900539] *** VALIDATE nfs4 *** [ 7.908137] pstore: using deflate compression [ 7.929390] Platform Keyring initialized [ 8.348856] NET: Registered protocol family 38 [ 8.350689] Key type asymmetric registered [ 8.352225] Asymmetric key parser 'x509' registered [ 8.361025] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 8.368228] io scheduler mq-deadline registered [ 8.370077] io scheduler kyber registered [ 8.374493] io scheduler bfq registered [ 8.380468] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 8.388583] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 8.393372] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 8.397844] ACPI: Power Button [PWRF] [ 8.409880] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 8.422872] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 8.443524] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 8.454916] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 8.490217] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 8.517302] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 8.546602] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 8.552989] Non-volatile memory driver v1.3 [ 8.555336] Linux agpgart interface v0.103 [ 8.591272] virtio_blk virtio1: [vda] 146016 512-byte logical blocks (74.8 MB/71.3 MiB) [ 8.602297] vda: detected capacity change from 0 to 74760192 [ 8.643211] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 8.647498] vdb: detected capacity change from 0 to 1073741824 [ 8.683575] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 8.690704] vdc: detected capacity change from 0 to 2621440000 [ 8.713503] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 8.722152] vdd: detected capacity change from 0 to 2621440000 [ 8.776688] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.782864] vde: detected capacity change from 0 to 4294967296 [ 8.808773] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.812635] vdf: detected capacity change from 0 to 4294967296 [ 8.825964] libphy: Fixed MDIO Bus: probed [ 8.880194] usbcore: registered new interface driver usbserial_generic [ 8.883280] usbserial: USB Serial support registered for generic [ 8.885996] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 8.890536] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 8.892800] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 8.897065] mousedev: PS/2 mouse device common for all mice [ 8.900746] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 8.904720] rtc_cmos 00:05: RTC can wake from S4 [ 8.912209] rtc_cmos 00:05: registered as rtc0 [ 8.924820] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 8.925808] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 8.930772] intel_pstate: CPU model not supported [ 8.949143] hid: raw HID events driver (C) Jiri Kosina [ 8.952994] usbcore: registered new interface driver usbhid [ 8.954983] usbhid: USB HID core driver [ 8.957224] drop_monitor: Initializing network drop monitor service [ 8.960296] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 8.961933] Initializing XFRM netlink socket [ 8.962636] NET: Registered protocol family 10 [ 8.964549] Segment Routing with IPv6 [ 8.964590] NET: Registered protocol family 17 [ 8.967156] mpls_gso: MPLS GSO support [ 8.984037] RAS: Correctable Errors collector initialized. [ 8.986878] AVX version of gcm_enc/dec engaged. [ 8.989235] AES CTR mode by8 optimization enabled [ 9.106260] sched_clock: Marking stable (9106237832, 0)->(10275004238, -1168766406) [ 9.112814] registered taskstats version 1 [ 9.122780] Loading compiled-in X.509 certificates [ 9.126021] zswap: loaded using pool lzo/zbud [ 9.173253] Key type big_key registered [ 9.196543] Key type encrypted registered [ 9.198587] ima: No TPM chip found, activating TPM-bypass! [ 9.204830] ima: Allocated hash algorithm: sha1 [ 9.207256] ima: No architecture policies found [ 9.209191] evm: Initialising EVM extended attributes: [ 9.212269] evm: security.selinux [ 9.217567] evm: security.ima [ 9.220273] evm: security.capability [ 9.222501] evm: HMAC attrs: 0x1 [ 9.226151] rtc_cmos 00:05: setting system clock to 2026-08-21 18:52:44 UTC (1787338364) [ 9.235921] debug: unmapping init [mem 0xffffffff86003000-0xffffffff861fffff] [ 9.240222] debug: unmapping init [mem 0xffffffff84d82000-0xffffffff85058fff] [ 9.248145] Write protecting the kernel read-only data: 28672k [ 9.253500] debug: unmapping init [mem 0xffffffff83403000-0xffffffff835fffff] [ 9.260670] debug: unmapping init [mem 0xffffffff83d14000-0xffffffff83dfffff] [ 9.375253] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 9.401939] systemd[1]: Detected virtualization kvm. [ 9.408673] systemd[1]: Detected architecture x86-64. [ 9.415394] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 9.473661] systemd[1]: No hostname configured. [ 9.478894] systemd[1]: Set hostname to . [ 9.484307] random: systemd: uninitialized urandom read (16 bytes read) [ 9.490312] systemd[1]: Initializing machine ID from random generator. [ 9.657236] random: ln: uninitialized urandom read (6 bytes read) [ 9.824755] random: systemd: uninitialized urandom read (16 bytes read) [ 9.829075] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 9.837526] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 9.844265] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Journal Service... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 11.561674] device-mapper: uevent: version 1.0.3 [ 11.571389] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 13.437339] virtio_net virtio0 ens2: renamed from eth0 [ 13.520106] random: fast init done [ 13.832306] scsi host0: ata_piix [ 13.846317] scsi host1: ata_piix [ 13.851530] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 13.866276] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 18.733982] random: crng init done [ 18.736871] random: 7 urandom warning(s) missed due to ratelimiting [ 20.925683] dracut-initqueue[582]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 23.569853] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 26.973959] printk: systemd: 26 output lines suppressed due to ratelimiting [ 27.896051] SELinux: Disabled at runtime. [ 28.067612] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 28.092172] systemd[1]: Detected virtualization kvm. [ 28.095340] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 29.512599] systemd[1]: initrd-switch-root.service: Succeeded. [ 29.516864] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 29.522941] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 29.527175] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 29.531453] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 29.546433] systemd[1]: Starting Journal Service... Starting Journal Service... [ 29.562837] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [ 30.097387] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 30.976641] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 32.063354] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 32.350990] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 32.787380] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 32.879582] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 38.610481] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 39.781695] NFS: Registering the id_resolver key type [ 39.796304] Key type id_resolver registered [ 39.802866] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (10s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg239-server login: [ 105.511601] libcfs: loading out-of-tree module taints kernel. [ 105.582776] Key type ._llcrypt registered [ 105.585661] Key type .llcrypt registered [ 105.686716] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 125.673738] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 127.540557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 127.574239] alg: No test for adler32 (adler32-zlib) [ 128.981708] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [ 130.042968] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 131.823273] Key type lgssc registered [ 133.884760] Lustre: Echo OBD driver; http://www.lustre.org/ [ 152.947563] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 198.285621] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 211.344466] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 211.364149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 212.659432] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 212.674683] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 212.782361] Lustre: lustre-MDT0000: new disk, initializing [ 212.865214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 212.878438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 217.121304] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 230.123758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 230.289123] Lustre: 6511:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 230.323178] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 230.326783] Lustre: Skipped 1 previous similar message [ 230.346064] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 230.414873] Lustre: lustre-MDT0001: new disk, initializing [ 230.506897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 230.552152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 230.558258] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 235.717964] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 241.507782] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 250.970938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 251.247964] Lustre: lustre-OST0000: new disk, initializing [ 251.250634] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 251.257719] Lustre: 8447:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 251.330779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 253.627735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 253.641400] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 253.713108] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 257.256199] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 271.816806] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 271.932781] Lustre: lustre-OST0001: new disk, initializing [ 271.941665] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 271.950724] Lustre: 9521:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 271.999367] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 278.442451] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 282.247768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 282.268720] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 282.424112] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 290.845876] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 297.283552] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 306.047555] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing check_logdir /tmp/testlogs/ [ 311.622405] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing yml_node [ 317.670459] Lustre: DEBUG MARKER: Client: 2.17.57.44 [ 321.091473] Lustre: DEBUG MARKER: MDS: 2.17.57.44 [ 324.326288] Lustre: DEBUG MARKER: OSS: 2.17.57.44 [ 326.764699] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Aug 21 14:58:00 EDT 2026 [ 347.516500] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 350.459598] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 368.614291] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 368.617903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 368.645750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 369.632085] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 369.658411] Lustre: Skipped 2 previous similar messages [ 369.664987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 369.669701] Lustre: Skipped 3 previous similar messages [ 373.971986] Lustre: server umount lustre-MDT0000 complete [ 377.029721] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787338732 with bad export cookie 3142668751182229643 [ 377.030336] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 377.045880] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 379.888337] LustreError: 6519:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 379.888384] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 379.896601] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 379.905247] LustreError: 6519:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 379.921698] Lustre: Skipped 1 previous similar message [ 382.883932] LustreError: 8454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 382.899437] LustreError: 8454:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 383.599125] Lustre: server umount lustre-MDT0001 complete [ 387.789437] Lustre: server umount lustre-OST0000 complete [ 391.946746] Lustre: server umount lustre-OST0001 complete [ 398.472805] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 408.894479] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 446.578820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 446.801806] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 446.827529] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 446.900398] Lustre: lustre-MDT0000: new disk, initializing [ 446.953931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 446.968480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 451.177100] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 460.746775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 460.864482] Lustre: 15076:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 460.907710] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 460.909922] Lustre: Skipped 1 previous similar message [ 460.960399] Lustre: lustre-MDT0001: new disk, initializing [ 461.045477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 461.082234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 461.090989] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 465.925680] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 474.726176] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 480.939741] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 486.784695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 487.084312] Lustre: lustre-OST0000: new disk, initializing [ 487.092387] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 487.108307] Lustre: 17041:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 487.198192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 489.045293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 489.055395] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 489.263736] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 493.543443] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 502.097084] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 506.924431] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 507.122841] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 511.127991] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 511.283379] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 515.052771] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 515.053364] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 515.060563] LustreError: Skipped 1 previous similar message [ 515.075744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 515.078415] Lustre: Skipped 1 previous similar message [ 515.112346] Lustre: Skipped 4 previous similar messages [ 525.283959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 525.292552] Lustre: Skipped 3 previous similar messages [ 527.327136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 527.485171] Lustre: server umount lustre-OST0000 complete [ 532.970119] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 532.978808] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 545.764293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 545.770064] Lustre: Skipped 6 previous similar messages [ 546.276446] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 548.606751] Lustre: server umount lustre-MDT0000 complete [ 550.897985] LustreError: 15087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 550.925667] LustreError: 15087:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 552.697320] LustreError: 17052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787338907 with bad export cookie 3142668751182231176 [ 552.698768] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 552.706042] LustreError: 17052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 553.037288] Lustre: server umount lustre-MDT0001 complete [ 557.579746] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 15:01:51 (1787338911) [ 560.435518] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 562.768809] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 15:01:57 (1787338917) [ 569.561514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 569.974888] LustreError: 19469:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 570.029335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 574.084512] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 575.458914] LustreError: 19470:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 575.481566] LustreError: 19470:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 581.735390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 586.574962] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 594.264852] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 600.050559] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 606.107582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 612.343491] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 620.434888] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 631.267339] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 631.282306] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 631.312737] Lustre: Skipped 1 previous similar message [ 631.330439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 631.339549] Lustre: Skipped 1 previous similar message [ 634.752357] Lustre: server umount lustre-OST0000 complete [ 646.088806] Lustre: server umount lustre-MDT0000 complete [ 646.624670] LustreError: 19464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 646.641201] LustreError: 19464:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 649.957862] LustreError: 19451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787339005 with bad export cookie 3142668751182232058 [ 649.971585] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 650.614852] Lustre: server umount lustre-MDT0001 complete [ 662.652465] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 665.332421] Key type lgssc unregistered [ 665.563430] LNet: 23004:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 665.579697] LNetError: 23004:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 665.593504] LNet: Removed LNI 192.168.202.139@tcp [ 666.603296] Key type .llcrypt unregistered [ 666.606457] Key type ._llcrypt unregistered [ 691.626345] Key type ._llcrypt registered [ 691.633210] Key type .llcrypt registered [ 691.756764] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 692.927765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 692.968302] alg: No test for adler32 (adler32-zlib) [ 694.595664] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [ 694.812542] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 696.511167] Key type lgssc registered [ 698.340202] Lustre: Echo OBD driver; http://www.lustre.org/ [ 707.920313] loop: module loaded [ 707.985495] loop0: detected capacity change from 0 to 204800000 [ 708.169714] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 708.176635] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 708.413339] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 715.343830] loop0: detected capacity change from 0 to 204800000 [ 715.490980] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 715.504661] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 715.529673] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 721.646940] loop0: detected capacity change from 0 to 204800000 [ 727.894619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 727.927567] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 729.212674] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 729.296436] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 729.359895] Lustre: lustre-MDT0000: new disk, initializing [ 729.434985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 729.465690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 734.202160] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 743.949964] loop1: detected capacity change from 0 to 204800000 [ 749.747760] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 750.318987] Lustre: lustre-OST07c6: new disk, initializing [ 750.328381] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 750.337754] Lustre: Skipped 1 previous similar message [ 750.348310] Lustre: 26194:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 750.431231] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 757.847443] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 757.854067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 757.867127] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 757.907768] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 762.963695] Lustre: 25182:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 762.972996] Lustre: 25182:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 762.981335] Lustre: 25182:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 762.989328] Lustre: 25182:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 762.999502] Lustre: 25182:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 763.007980] Lustre: 25182:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 780.107877] Lustre: 25182:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 780.117474] Lustre: 25182:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 780.126661] Lustre: 25182:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 780.131223] Lustre: 25182:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 780.137603] Lustre: 25182:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 780.146404] Lustre: 25182:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 785.105350] Lustre: 25183:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 785.115305] Lustre: 25183:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 785.122744] Lustre: 25183:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 785.126991] Lustre: 25183:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 785.133626] Lustre: 25183:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 785.144555] Lustre: 25183:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 785.148392] Lustre: 25183:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 785.154206] Lustre: 25183:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 785.159775] Lustre: 25183:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 785.164609] Lustre: 25183:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 785.173432] Lustre: 25183:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 785.179087] Lustre: 25183:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 788.431753] Lustre: 25183:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 788.437945] Lustre: 25183:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 788.445904] Lustre: 25183:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 788.456704] Lustre: 25183:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 788.462586] Lustre: 25183:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 788.467551] Lustre: 25183:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 788.472941] Lustre: 25183:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 788.478840] Lustre: 25183:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 788.484397] Lustre: 25183:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 788.489261] Lustre: 25183:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 788.494425] Lustre: 25183:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 788.500420] Lustre: 25183:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 793.348344] Lustre: 25183:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 793.377306] Lustre: 25183:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 793.392232] Lustre: 25183:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 793.408157] Lustre: 25183:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 793.417118] Lustre: 25183:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 793.431739] Lustre: 25183:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 793.442165] Lustre: 25183:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 793.456042] Lustre: 25183:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 793.468069] Lustre: 25183:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 793.476209] Lustre: 25183:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 793.485441] Lustre: 25183:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 793.498230] Lustre: 25183:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 805.125555] Lustre: 26883:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 805.141145] Lustre: 26883:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 805.149830] Lustre: 26883:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 805.155726] Lustre: 26883:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 805.163164] Lustre: 26883:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 805.169593] Lustre: 26883:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 805.174973] Lustre: 26883:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 805.183199] Lustre: 26883:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 805.189821] Lustre: 26883:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 805.195832] Lustre: 26883:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 805.201347] Lustre: 26883:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 805.206501] Lustre: 26883:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 855.019765] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 855.024294] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 855.032880] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 858.742974] Lustre: server umount lustre-OST07c6 complete [ 867.925692] Lustre: server umount lustre-MDT0000 complete [ 884.707781] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 887.848699] Key type lgssc unregistered [ 888.091960] LNet: 28758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 888.104411] LNetError: 28758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 889.137083] LNet: Removed LNI 192.168.202.139@tcp [ 890.073795] Key type .llcrypt unregistered [ 890.075381] Key type ._llcrypt unregistered [ 911.131396] Key type ._llcrypt registered [ 911.136500] Key type .llcrypt registered [ 911.277830] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 912.432817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 912.542803] alg: No test for adler32 (adler32-zlib) [ 913.715838] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [ 914.089234] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 915.751199] Key type lgssc registered [ 916.908915] Lustre: Echo OBD driver; http://www.lustre.org/ [ 925.625830] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 925.642985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 927.094672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 932.689708] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 937.443986] LustreError: 30033:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 937.458787] LustreError: 30033:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 942.565607] LustreError: 30034:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 943.046765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 943.479324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 949.500209] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 958.791632] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 965.196146] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 973.182826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 973.814640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 981.219377] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 990.038319] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 997.380233] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1017.836361] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1017.852253] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1017.856701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1017.860154] Lustre: Skipped 1 previous similar message [ 1022.002806] Lustre: server umount lustre-OST0000 complete [ 1028.071421] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1028.073755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1028.093508] Lustre: Skipped 1 previous similar message [ 1028.110061] Lustre: Skipped 2 previous similar messages [ 1033.185207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1033.199246] Lustre: Skipped 1 previous similar message [ 1038.307384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1038.315045] Lustre: Skipped 1 previous similar message [ 1041.376933] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1041.553571] Lustre: server umount lustre-MDT0000 complete [ 1043.424691] LustreError: 30761:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1043.440703] LustreError: 30761:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1045.296100] LustreError: 33041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787339400 with bad export cookie 10150438019371656723 [ 1045.301752] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1045.308739] LustreError: 33041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1045.788615] Lustre: server umount lustre-MDT0001 complete [ 1057.908665] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 1061.305845] Key type lgssc unregistered [ 1061.649784] LNet: 33798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1061.660304] LNetError: 33798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1061.698824] LNet: Removed LNI 192.168.202.139@tcp [ 1062.702422] Key type .llcrypt unregistered [ 1062.704520] Key type ._llcrypt unregistered [ 1083.651734] Key type ._llcrypt registered [ 1083.657099] Key type .llcrypt registered [ 1083.794682] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 1085.249051] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1085.576842] alg: No test for adler32 (adler32-zlib) [ 1086.873408] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [ 1087.034656] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 1088.679307] Key type lgssc registered [ 1090.341256] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1098.016654] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1098.038463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1099.512142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1104.469269] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1108.965655] LustreError: 35070:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1109.010830] LustreError: 35070:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1114.082302] LustreError: 35069:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1114.670613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1114.995445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1119.806682] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1128.048868] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1134.478514] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1141.314836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1141.657373] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1144.772748] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1146.920503] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1155.211575] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1170.404409] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1170.416387] Lustre: Skipped 1 previous similar message [ 1170.418437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1170.421511] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1170.424761] Lustre: Skipped 1 previous similar message [ 1170.433412] LustreError: Skipped 1 previous similar message [ 1174.152866] Lustre: server umount lustre-OST0000 complete [ 1180.646070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1180.666538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1184.284821] Lustre: server umount lustre-MDT0000 complete [ 1185.763676] LustreError: 35070:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1185.782520] LustreError: 35070:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1188.470325] LustreError: 35048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787339543 with bad export cookie 5640604047520275280 [ 1188.478969] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1188.482368] LustreError: 35048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1188.931447] Lustre: server umount lustre-MDT0001 complete [ 1201.707958] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 1204.650373] Key type lgssc unregistered [ 1204.861905] LNet: 38627:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1204.877853] LNetError: 38627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1204.900805] LNet: Removed LNI 192.168.202.139@tcp [ 1205.723758] Key type .llcrypt unregistered [ 1205.725538] Key type ._llcrypt unregistered [ 1230.963199] Key type ._llcrypt registered [ 1230.966512] Key type .llcrypt registered [ 1231.083387] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 1232.010371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1232.087951] alg: No test for adler32 (adler32-zlib) [ 1233.297786] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [ 1233.540315] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 1235.207329] Key type lgssc registered [ 1236.398872] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1244.873707] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1244.895057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1246.542846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1251.181643] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1255.970978] LustreError: 40044:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1255.988872] LustreError: 40044:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1259.944065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1260.208512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1265.604872] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1273.978772] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1280.707803] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1287.820679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1288.262855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1291.393686] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1295.776047] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1304.313981] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1311.712818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1311.716320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1311.731312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1311.996081] Lustre: server umount lustre-MDT0000 complete [ 1312.534210] LustreError: 40041:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1315.983676] LustreError: 40025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787339671 with bad export cookie 1651164870093972853 [ 1315.991184] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1315.999166] LustreError: 40025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1316.320947] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1316.341327] LustreError: 41699:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1316.345544] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1317.618810] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.39@tcp (stopping) [ 1320.352715] LustreError: 42536:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1320.362839] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1320.365841] LustreError: 42536:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1322.486695] Lustre: server umount lustre-MDT0001 complete [ 1343.660177] Lustre: server umount lustre-OST0000 complete [ 1361.140581] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 1364.122637] Key type lgssc unregistered [ 1364.541577] LNet: 43908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1364.553117] LNetError: 43908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1364.582943] LNet: Removed LNI 192.168.202.139@tcp [ 1365.286253] Key type .llcrypt unregistered [ 1365.288607] Key type ._llcrypt unregistered [ 1393.095339] Key type ._llcrypt registered [ 1393.097956] Key type .llcrypt registered [ 1393.273658] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 1394.537551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1394.672380] alg: No test for adler32 (adler32-zlib) [ 1395.808774] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [ 1396.047677] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 1397.712048] Key type lgssc registered [ 1398.757404] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1407.057546] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1407.090313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1408.564838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1413.780205] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1418.727926] LustreError: 45326:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1418.750722] LustreError: 45326:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1423.844357] LustreError: 45327:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1425.066647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1425.445991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1430.848523] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1438.685076] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1445.448899] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1453.292294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1453.648935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1460.192844] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1468.861654] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1478.195337] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1488.357697] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1488.366809] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1488.399250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1491.203204] Lustre: lustre-OST0000: Not available for connect from 192.168.202.39@tcp (stopping) [ 1491.213667] Lustre: Skipped 1 previous similar message [ 1493.391494] Lustre: server umount lustre-OST0000 complete [ 1512.931311] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1512.939931] Lustre: Skipped 1 previous similar message [ 1512.945271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1517.537050] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1517.550173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1517.568905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1523.168834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1523.179860] Lustre: Skipped 3 previous similar messages [ 1527.263532] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1527.471134] Lustre: server umount lustre-MDT0000 complete [ 1528.294580] LustreError: 45327:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1528.321565] LustreError: 45327:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1533.173514] LustreError: 45307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787339888 with bad export cookie 10701085251785960416 [ 1533.185107] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1533.188276] LustreError: 45307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1539.724122] Lustre: server umount lustre-MDT0001 complete [ 1554.921275] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 1558.600595] Key type lgssc unregistered [ 1558.988760] LNet: 49068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1559.000602] LNetError: 49068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1559.025617] LNet: Removed LNI 192.168.202.139@tcp [ 1560.213071] Key type .llcrypt unregistered [ 1560.219746] Key type ._llcrypt unregistered [ 1586.602794] Key type ._llcrypt registered [ 1586.608360] Key type .llcrypt registered [ 1586.780794] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 1588.145481] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1588.175419] alg: No test for adler32 (adler32-zlib) [ 1589.234872] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [ 1589.595049] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 1591.352068] Key type lgssc registered [ 1592.707917] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1601.307267] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1601.334335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1602.768470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1607.498691] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1613.294731] LustreError: 50493:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1613.314168] LustreError: 50493:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1616.775978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1617.405286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1622.106342] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1630.714342] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1636.353748] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1643.736841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1644.129721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1648.238825] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1650.177254] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1657.862702] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1667.534859] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1670.932885] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1682.967607] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-2115h20m37s [ 1684.538199] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1688.546973] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1688.553557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1688.568592] Lustre: Skipped 1 previous similar message [ 1688.592216] Lustre: Skipped 6 previous similar messages [ 1689.644053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1689.648501] Lustre: Skipped 1 previous similar message [ 1691.376102] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.39@tcp (stopping) [ 1691.384809] Lustre: Skipped 1 previous similar message [ 1693.152434] Lustre: server umount lustre-MDT0000 complete [ 1695.714880] LustreError: 50493:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1695.744561] LustreError: 50493:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1697.276617] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1698.524652] LustreError: 50489:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1701.883597] LustreError: 52845:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1701.905134] LustreError: 52845:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1706.161058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1706.393584] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1706.873715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1706.985496] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1707.880344] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1713.746338] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1720.124567] Lustre: DEBUG MARKER: Wait for df (36728) ... [ 1722.926764] Lustre: DEBUG MARKER: done [ 1732.577520] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1732.592474] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1732.613718] Lustre: Skipped 1 previous similar message [ 1732.621271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1732.634017] Lustre: Skipped 1 previous similar message [ 1733.602302] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1738.736511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1738.752106] Lustre: Skipped 3 previous similar messages [ 1745.377216] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1745.801971] Lustre: server umount lustre-OST0000 complete [ 1754.081975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1754.084291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1754.086211] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1754.101807] Lustre: Skipped 1 previous similar message [ 1754.149885] Lustre: Skipped 3 previous similar messages [ 1757.631303] Lustre: server umount lustre-MDT0000 complete [ 1761.255602] LustreError: 52845:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1761.276135] LustreError: 52845:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1761.466917] LustreError: 50473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787340116 with bad export cookie 11666010539522193888 [ 1761.467715] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1761.479446] LustreError: 50473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1761.925274] Lustre: server umount lustre-MDT0001 complete [ 1773.705224] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 1776.340466] Key type lgssc unregistered [ 1776.577410] LNet: 55425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1776.584361] LNetError: 55425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1776.597318] LNet: Removed LNI 192.168.202.139@tcp [ 1777.283458] Key type .llcrypt unregistered [ 1777.287480] Key type ._llcrypt unregistered [ 1816.628921] Key type ._llcrypt registered [ 1816.631295] Key type .llcrypt registered [ 1816.753983] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 1818.044186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1818.086101] alg: No test for adler32 (adler32-zlib) [ 1819.186875] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [ 1819.427524] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 1821.079345] Key type lgssc registered [ 1822.178883] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1829.788872] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1829.831569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1831.123507] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1831.174486] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1831.392518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1835.622893] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1843.841289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1843.976114] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1844.027930] Lustre: 57636:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1844.306687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1849.452845] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1856.918707] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1862.135993] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1869.035601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1869.243532] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1869.435099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1875.154136] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1875.920746] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1884.391917] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1892.765752] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-2115h24m06s [ 1894.730689] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1902.994250] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1905.810543] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1905.933841] LustreError: 57643:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1926.449453] Lustre: lustre-MDT0000: Client 990c4990-efbe-4101-8ff4-ea5705b54a22 (at 192.168.202.39@tcp) reconnecting [ 1946.031143] LustreError: 57643:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1948.663857] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1951.682174] Lustre: DEBUG MARKER: done [ 1957.275566] Lustre: server umount lustre-OST0000 complete [ 1961.952205] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1961.964189] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1961.976901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1963.040518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1963.045573] Lustre: Skipped 2 previous similar messages [ 1969.121684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1969.127060] Lustre: Skipped 1 previous similar message [ 1971.266679] Lustre: server umount lustre-MDT0000 complete [ 1973.215746] Lustre: 56956:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787340312/real 1787340312] req@00000000f451c123 x1874162010347904/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787340328 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1973.218969] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1973.250304] Lustre: 56956:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1973.284900] Lustre: Skipped 1 previous similar message [ 1974.464045] LustreError: 60395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787340329 with bad export cookie 10845572768639660958 [ 1974.474169] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1974.475873] LustreError: 60395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1974.783627] Lustre: server umount lustre-MDT0001 complete [ 1986.937773] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 1989.913740] Key type lgssc unregistered [ 1990.264767] LNet: 61968:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1990.269739] LNetError: 61968:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1990.294715] LNet: Removed LNI 192.168.202.139@tcp [ 1991.243694] Key type .llcrypt unregistered [ 1991.246928] Key type ._llcrypt unregistered [ 2039.988944] Key type ._llcrypt registered [ 2039.991573] Key type .llcrypt registered [ 2040.173916] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 2042.211229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2042.267555] alg: No test for adler32 (adler32-zlib) [ 2043.659159] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [ 2043.932138] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 2045.632323] Key type lgssc registered [ 2047.912275] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2053.238083] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2053.268421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2054.573976] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2054.612081] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2054.794728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2055.898864] Lustre: Failing over lustre-MDT0000 [ 2056.139884] Lustre: server umount lustre-MDT0000 complete [ 2065.684023] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 15:26:59 (1787340419) [ 2074.816198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2075.442043] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2081.597381] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2090.692348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2090.762978] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2090.793555] Lustre: 64770:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2091.020255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2096.152974] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2103.765906] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2109.424250] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2116.072900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2116.289861] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2116.598835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2122.737511] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2123.384896] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2130.028602] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2136.365558] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2144.235700] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2144.241229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2144.265855] Lustre: Skipped 2 previous similar messages [ 2144.273696] Lustre: Skipped 1 previous similar message [ 2148.958043] Lustre: server umount lustre-MDT0000 complete [ 2152.462763] LustreError: 64762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787340507 with bad export cookie 3230403734219375523 [ 2152.471071] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2152.478701] LustreError: 64762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2153.080451] Lustre: server umount lustre-MDT0001 complete [ 2157.075398] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2162.540416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2169.825615] Lustre: 63688:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787340509/real 1787340509] req@000000007946c4ea x1874162246262784/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787340525 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2169.893438] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2172.963357] Lustre: 63690:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787340512/real 1787340512] req@000000002cef6d46 x1874162246262912/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787340528 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2173.994952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2174.943220] Lustre: 63691:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787340514/real 1787340514] req@00000000728b09d2 x1874162246263040/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787340530 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2177.122848] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x2cd4b314b33dcc86 [ 2177.140221] Lustre: MGC192.168.202.139@tcp: Connection restored to 0@lo (at 0@lo) [ 2177.502091] LustreError: 66617:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2177.519678] LustreError: 66617:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2177.623289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2177.763597] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2178.592105] Lustre: 63688:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787340517/real 1787340517] req@000000007aa612ab x1874162246263296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787340533 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2179.620234] LustreError: 66618:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2180.642704] LustreError: 68540:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2180.667440] LustreError: 68540:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2183.048455] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2184.673466] Lustre: 63688:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787340523/real 1787340523] req@000000003308bbd6 x1874162246263424/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787340539 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2185.697637] LustreError: 66617:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2185.724481] LustreError: 66617:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2190.824756] LustreError: 68541:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2190.848882] LustreError: 68541:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2194.108216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2194.578030] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2199.733610] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2200.152978] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2210.194672] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2216.796614] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2225.639195] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2225.652528] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2225.657857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2225.671931] Lustre: Skipped 1 previous similar message [ 2230.573725] Lustre: server umount lustre-MDT0000 complete [ 2230.763074] LustreError: 66617:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2230.813283] LustreError: 66617:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2234.769042] LustreError: 64761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787340590 with bad export cookie 3230403734219377798 [ 2234.771826] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2234.780496] LustreError: 64761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2235.385497] Lustre: server umount lustre-MDT0001 complete [ 2242.771933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2252.259689] Lustre: 63690:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787340591/real 1787340591] req@000000003636422e x1874162246291072/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787340607 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2252.302388] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2252.317075] Lustre: Skipped 1 previous similar message [ 2253.464235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2260.847816] LustreError: 69275:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2260.910431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2260.966424] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2265.899237] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2277.063031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2281.138530] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2281.148140] Lustre: Skipped 1 previous similar message [ 2284.594759] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2293.357458] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2300.290877] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2308.578427] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2308.593382] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2308.607580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2308.614221] Lustre: Skipped 2 previous similar messages [ 2312.162650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2313.688305] Lustre: server umount lustre-MDT0000 complete [ 2317.125860] LustreError: 65513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787340672 with bad export cookie 3230403734219379177 [ 2317.127773] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2317.138253] LustreError: 65513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2317.293285] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2317.299481] Lustre: Skipped 2 previous similar messages [ 2317.303832] LustreError: 71224:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2317.311622] LustreError: 71224:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 2317.439295] Lustre: server umount lustre-MDT0001 complete [ 2324.395490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2328.269656] Lustre: DEBUG MARKER: files compared the same [ 2330.603421] Lustre: server umount lustre-OST0000 complete [ 2350.755349] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 2353.859330] Key type lgssc unregistered [ 2354.321165] LNet: 74503:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2354.327374] LNetError: 74503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2354.342889] LNet: Removed LNI 192.168.202.139@tcp [ 2355.221630] Key type .llcrypt unregistered [ 2355.224309] Key type ._llcrypt unregistered [ 2385.061961] Key type ._llcrypt registered [ 2385.073163] Key type .llcrypt registered [ 2385.192237] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 2386.381352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2386.630886] alg: No test for adler32 (adler32-zlib) [ 2387.742342] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [ 2388.040607] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 2389.775205] Key type lgssc registered [ 2391.239632] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2399.300295] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2399.313425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2400.799662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2405.768536] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug +malloc all [ 2410.988101] LustreError: 75921:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2411.016649] LustreError: 75921:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2414.829152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2415.241654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2420.756157] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug +malloc all [ 2429.573829] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2436.497793] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2444.810467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2445.132086] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2446.205278] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2451.890546] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug +malloc all [ 2460.938740] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2470.881187] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2470.887511] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2470.895490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2470.901983] Lustre: Skipped 1 previous similar message [ 2475.137767] Lustre: server umount lustre-OST0000 complete [ 2481.120052] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2481.122888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2481.138136] Lustre: Skipped 1 previous similar message [ 2481.154017] Lustre: Skipped 2 previous similar messages [ 2484.885939] Lustre: server umount lustre-MDT0000 complete [ 2486.239611] LustreError: 75921:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2486.258753] LustreError: 75921:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2488.774769] LustreError: 75902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787340844 with bad export cookie 4631067358588120452 [ 2488.788162] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2489.216911] Lustre: server umount lustre-MDT0001 complete [ 2500.600079] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 2503.318264] Key type lgssc unregistered [ 2503.624733] LNet: 79458:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2503.631878] LNetError: 79458:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2503.650800] LNet: Removed LNI 192.168.202.139@tcp [ 2504.558158] Key type .llcrypt unregistered [ 2504.561193] Key type ._llcrypt unregistered [ 2534.171522] Key type ._llcrypt registered [ 2534.173201] Key type .llcrypt registered [ 2534.274078] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 2535.443400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2535.628993] alg: No test for adler32 (adler32-zlib) [ 2536.659339] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [ 2536.822393] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 2538.439467] Key type lgssc registered [ 2539.571646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2547.225647] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2547.239396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2562.719563] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2567.839454] LustreError: 80854:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.139@tcp: failed processing log, type 4: rc = -110 [ 2594.655721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2600.784056] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2607.458794] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2615.177460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2615.284658] Lustre: *** cfs_fail_loc=706, val=0*** [ 2615.732448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2615.799117] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2619.913681] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2620.907268] LustreError: 80880:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2620.938958] LustreError: 80880:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2626.018947] LustreError: 80879:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2629.113756] LustreError: 80880:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2629.141091] LustreError: 80880:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2629.166959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2629.512577] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2633.964802] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2641.870866] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2647.251639] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2650.100599] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2650.107934] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2650.141225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2650.154664] Lustre: Skipped 1 previous similar message [ 2655.212352] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2655.215465] Lustre: Skipped 1 previous similar message [ 2655.595518] Lustre: server umount lustre-OST0000 complete [ 2660.320323] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2660.323652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2660.336621] Lustre: Skipped 2 previous similar messages [ 2660.356876] Lustre: Skipped 1 previous similar message [ 2665.441347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2665.458742] Lustre: Skipped 1 previous similar message [ 2665.918509] Lustre: server umount lustre-MDT0000 complete [ 2669.274975] LustreError: 82677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787341024 with bad export cookie 9067304303500850094 [ 2669.290525] LustreError: 82677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2669.292073] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2669.612137] Lustre: server umount lustre-MDT0001 complete [ 2680.084947] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 2682.521529] Key type lgssc unregistered [ 2682.746643] LNet: 84470:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2682.751067] LNetError: 84470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2682.773837] LNet: Removed LNI 192.168.202.139@tcp [ 2683.472494] Key type .llcrypt unregistered [ 2683.474541] Key type ._llcrypt unregistered [ 2707.572027] Key type ._llcrypt registered [ 2707.577342] Key type .llcrypt registered [ 2707.713675] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 2708.853928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2709.011596] alg: No test for adler32 (adler32-zlib) [ 2710.207645] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [ 2710.499105] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 2712.165362] Key type lgssc registered [ 2713.836879] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2721.606711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2721.633233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2727.720700] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2735.536033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2736.027634] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2740.598432] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2741.236526] LustreError: 86574:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2741.264733] LustreError: 86574:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2746.337641] LustreError: 86577:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2746.362230] LustreError: 86577:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2747.385553] LustreError: 86578:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2747.404698] LustreError: 86578:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2749.786328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2750.206533] LustreError: 86572:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2750.239214] LustreError: 86572:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2750.262384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2755.553328] LustreError: 86577:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2755.600722] LustreError: 86577:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2756.878521] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2766.645624] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2766.676575] Lustre: MGS: Client 0c7b3bf3-77b1-4a87-8266-2a55fbdcbbb9 (at 0@lo) reconnecting [ 2766.707884] LustreError: 86588:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2766.989787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2767.123703] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2771.906835] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2787.817906] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2787.838361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2792.419537] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2792.436459] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2792.460839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2793.803865] Lustre: server umount lustre-OST0000 complete [ 2798.050140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2798.053355] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2798.059502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2798.059510] Lustre: Skipped 1 previous similar message [ 2798.073778] Lustre: Skipped 1 previous similar message [ 2803.169239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2803.174620] Lustre: Skipped 2 previous similar messages [ 2803.560378] Lustre: server umount lustre-MDT0000 complete [ 2808.007235] Lustre: server umount lustre-MDT0001 complete [ 2811.794819] Lustre: server umount MGS complete [ 2823.314903] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 15:39:37 (1787341177) [ 2855.912980] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 2868.555433] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 2908.249305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2913.071105] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2923.603094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2923.896263] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2923.979519] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2924.090973] Lustre: lustre-MDT0001: new disk, initializing [ 2924.232100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2924.254514] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2924.265869] Lustre: 93768:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2924.276600] LustreError: 93768:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2929.417266] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2940.183444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2940.722653] Lustre: lustre-OST0000: new disk, initializing [ 2940.728464] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2940.740962] Lustre: 94533:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2940.843772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2949.142592] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2951.776355] Lustre: 94969:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000062c45639 x1874162944614272/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787341323 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2958.347133] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2966.044804] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2966.063383] Lustre: MGS: Client 3d0e3faa-6083-42a3-972d-c7595a76bec1 (at 0@lo) reconnecting [ 2966.087136] LustreError: 93754:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2966.139498] Lustre: 93036:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2966.205921] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2966.325547] Lustre: lustre-MDT0000: new disk, initializing [ 2966.555848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2966.629373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2966.658514] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2967.727429] Lustre: 95602:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000057d468cd x1874162944621056/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787341338 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2971.338751] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2972.147526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2972.170331] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2972.855296] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2972.867635] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2997.730090] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2997.751748] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2997.753623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2997.792953] Lustre: Skipped 1 previous similar message [ 3007.970032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3007.975101] Lustre: Skipped 3 previous similar messages [ 3012.063195] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3012.280235] Lustre: server umount lustre-OST0000 complete [ 3018.209806] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3018.215197] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3018.238696] Lustre: Skipped 1 previous similar message [ 3022.628457] Lustre: server umount lustre-MDT0000 complete [ 3023.341452] LustreError: 93740:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3023.365875] LustreError: 93740:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 3033.141686] Lustre: server umount lustre-MDT0001 complete [ 3037.121677] Lustre: server umount MGS complete [ 3050.353268] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 15:43:24 (1787341404) [ 3068.443165] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 3071.316290] Key type lgssc unregistered [ 3071.583676] LNet: 98105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3071.599910] LNetError: 98105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3071.627897] LNet: Removed LNI 192.168.202.139@tcp [ 3072.384852] Key type .llcrypt unregistered [ 3072.387241] Key type ._llcrypt unregistered [ 3093.278937] Key type ._llcrypt registered [ 3093.282612] Key type .llcrypt registered [ 3093.409241] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 3094.956996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3095.093212] alg: No test for adler32 (adler32-zlib) [ 3096.119210] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [ 3096.326451] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 3097.983241] Key type lgssc registered [ 3099.442790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3106.870525] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3106.903784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3108.300443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3113.370567] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3118.571092] LustreError: 99325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3118.601059] LustreError: 99325:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3122.287326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3122.648944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3127.616161] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3136.238539] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3143.213439] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3150.741711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3151.132914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3153.212929] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 3156.883209] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3165.076492] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3177.954220] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3177.956157] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3177.960463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3177.985413] Lustre: Skipped 1 previous similar message [ 3182.790546] Lustre: server umount lustre-OST0000 complete [ 3188.194110] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3188.197538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3188.217449] Lustre: Skipped 1 previous similar message [ 3188.230497] Lustre: Skipped 2 previous similar messages [ 3192.690428] Lustre: server umount lustre-MDT0000 complete [ 3193.312904] LustreError: 99325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3193.341489] LustreError: 99325:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3196.819056] LustreError: 99306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787341552 with bad export cookie 17225398634985428773 [ 3196.830161] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3196.833627] LustreError: 99306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3197.118826] Lustre: server umount lustre-MDT0001 complete [ 3209.177085] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 3211.595694] Key type lgssc unregistered [ 3211.881390] LNet: 102861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3211.888627] LNetError: 102861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3211.905180] LNet: Removed LNI 192.168.202.139@tcp [ 3212.749785] Key type .llcrypt unregistered [ 3212.752298] Key type ._llcrypt unregistered [ 3232.318238] Key type ._llcrypt registered [ 3232.323805] Key type .llcrypt registered [ 3232.460856] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 3234.118250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3234.476410] alg: No test for adler32 (adler32-zlib) [ 3235.673915] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [ 3235.998283] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 3237.683167] Key type lgssc registered [ 3239.156112] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3245.426228] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3245.430807] LustreError: 103931:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3245.533957] LustreError: 103946:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3245.540713] LustreError: 103931:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4896 [ 3245.551318] LustreError: 103931:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3245.556815] LustreError: 103931:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3245.575188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3247.063462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3256.703635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3257.075687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3262.766494] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3269.685861] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing lsmod [ 3273.095053] LustreError: 105151:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3273.117127] LustreError: 105153:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3273.133127] LustreError: 105151:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4984 [ 3273.153365] LustreError: 105151:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3273.176073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3273.179578] LustreError: 105151:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3273.396631] LustreError: 105165:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3273.439878] LustreError: 105165:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3273.874862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3277.804066] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3277.821397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3277.845085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3279.335666] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3279.353647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3279.360953] Lustre: Skipped 1 previous similar message [ 3279.397172] Lustre: Skipped 2 previous similar messages [ 3282.466673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3282.481486] Lustre: Skipped 2 previous similar messages [ 3284.413642] Lustre: server umount lustre-MDT0000 complete [ 3287.527448] LustreError: 103978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3287.545647] LustreError: 103978:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3289.240325] LustreError: 103958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787341644 with bad export cookie 188361009741803455 [ 3289.250624] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3290.719315] LustreError: 104416:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000da17b7fc x1874163496146816/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3290.907484] Lustre: server umount lustre-MDT0001 complete [ 3305.596743] Lustre: server umount lustre-OST0000 complete [ 3313.637548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3314.193247] LustreError: 106218:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3314.331182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3319.694455] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3319.791742] LustreError: 106219:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3323.881509] LustreError: 106218:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3329.107718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3329.563120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3334.775402] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3342.494434] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3348.377571] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3355.863346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3356.387640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3360.509337] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3362.523709] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3369.333707] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3385.321968] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3385.326434] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3385.342797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3388.223349] Lustre: server umount lustre-OST0000 complete [ 3395.553806] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3395.557064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3395.569599] Lustre: Skipped 2 previous similar messages [ 3395.595671] Lustre: Skipped 2 previous similar messages [ 3404.771835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3404.782929] Lustre: Skipped 3 previous similar messages [ 3406.303151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3406.550685] Lustre: server umount lustre-MDT0000 complete [ 3409.897771] LustreError: 106219:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3409.924820] LustreError: 106219:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3410.335175] LustreError: 106201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787341765 with bad export cookie 188361009741804155 [ 3410.338935] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3410.352884] LustreError: 106201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3410.605677] Lustre: server umount lustre-MDT0001 complete [ 3422.605355] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 3425.619418] Key type lgssc unregistered [ 3425.986768] LNet: 109773:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3425.996126] LNetError: 109773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3427.044896] LNet: Removed LNI 192.168.202.139@tcp [ 3427.883925] Key type .llcrypt unregistered [ 3427.885576] Key type ._llcrypt unregistered [ 3456.404802] Key type ._llcrypt registered [ 3456.408901] Key type .llcrypt registered [ 3456.514787] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 3457.678659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3457.840736] alg: No test for adler32 (adler32-zlib) [ 3458.870761] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [ 3459.060346] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 3460.737273] Key type lgssc registered [ 3462.024269] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3469.427883] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3469.449855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3470.946246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3475.614292] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3481.058694] LustreError: 111189:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3481.078148] LustreError: 111189:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3483.597492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3484.035391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3488.504170] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3495.604251] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3501.074038] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3508.532517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3508.848249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3514.820723] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3516.020851] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 3521.954637] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3528.123767] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3531.979210] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3535.852049] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3535.860416] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3535.869960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3536.867684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3536.870741] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3536.873115] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3541.997097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3542.003038] Lustre: Skipped 2 previous similar messages [ 3547.130580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3547.166153] Lustre: Skipped 1 previous similar message [ 3550.175688] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3550.349094] Lustre: server umount lustre-OST0000 complete [ 3555.808771] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3555.823097] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3555.854558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3562.010813] Lustre: server umount lustre-MDT0000 complete [ 3562.468441] LustreError: 111190:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3562.486793] LustreError: 111190:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3565.919210] LustreError: 111171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787341921 with bad export cookie 7215579359519969312 [ 3565.924927] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3565.931992] LustreError: 111171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3566.284356] Lustre: server umount lustre-MDT0001 complete [ 3579.104875] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 3581.690418] Key type lgssc unregistered [ 3581.939912] LNet: 114831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3581.953084] LNetError: 114831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3581.970948] LNet: Removed LNI 192.168.202.139@tcp [ 3582.869256] Key type .llcrypt unregistered [ 3582.871766] Key type ._llcrypt unregistered [ 3600.397741] Key type ._llcrypt registered [ 3600.400665] Key type .llcrypt registered [ 3600.500757] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 3601.329181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3601.482615] alg: No test for adler32 (adler32-zlib) [ 3602.505257] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [ 3602.721715] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 3604.367334] Key type lgssc registered [ 3605.432336] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3613.638472] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3613.663493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3614.992050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3618.971538] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3625.449405] LustreError: 116062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3625.471760] LustreError: 116062:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3627.176471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3627.565992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3632.808357] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3640.328766] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3645.581321] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3652.199732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3652.440471] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3658.774546] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3660.669414] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3669.771421] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3681.251158] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3681.254972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3681.293306] Lustre: Skipped 1 previous similar message [ 3681.293811] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3681.329982] Lustre: Skipped 1 previous similar message [ 3686.013381] Lustre: server umount lustre-OST0000 complete [ 3691.498081] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3691.518145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3691.529349] Lustre: Skipped 1 previous similar message [ 3695.965996] Lustre: server umount lustre-MDT0000 complete [ 3696.617314] LustreError: 117026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3696.634925] LustreError: 117026:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3699.768616] LustreError: 116043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787342055 with bad export cookie 12923503099688146460 [ 3699.779024] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3699.780764] LustreError: 116043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3700.315206] Lustre: server umount lustre-MDT0001 complete [ 3712.335389] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 3714.615221] Key type lgssc unregistered [ 3714.874653] LNet: 119602:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3714.884675] LNetError: 119602:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3714.901590] LNet: Removed LNI 192.168.202.139@tcp [ 3715.610419] Key type .llcrypt unregistered [ 3715.613497] Key type ._llcrypt unregistered [ 3741.662353] Key type ._llcrypt registered [ 3741.664398] Key type .llcrypt registered [ 3741.748668] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 3742.789043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3743.010237] alg: No test for adler32 (adler32-zlib) [ 3744.179716] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [ 3744.360176] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 3746.023366] Key type lgssc registered [ 3747.190754] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3754.910461] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3754.924912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3756.608041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3760.859229] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3765.747992] LustreError: 121021:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3765.769504] LustreError: 121021:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3770.078542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3770.422374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3775.428598] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3783.780914] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3790.744424] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3798.678547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3799.250876] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3801.294941] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3806.470376] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3815.603039] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3823.939517] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3828.376403] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3828.382931] Lustre: Skipped 2 previous similar messages [ 3833.380665] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3833.384893] Lustre: Skipped 2 previous similar messages [ 3834.603379] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3834.675645] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3841.258600] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.39@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3841.273865] Lustre: Skipped 2 previous similar messages [ 3848.280459] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.39@tcp 192.168.0.[2,10]@tcp [ 3853.810868] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.39/24@tcp in log lustre-MDT0000 [ 3853.819224] Lustre: Skipped 2 previous similar messages [ 3856.859288] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.39/24@tcp [ 3856.868496] Lustre: Skipped 1 previous similar message [ 3863.015228] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3863.029932] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3863.041370] LustreError: Skipped 1 previous similar message [ 3863.061389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3867.108727] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3867.112073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3867.129401] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3867.169105] Lustre: Skipped 1 previous similar message [ 3868.960865] Lustre: server umount lustre-OST0000 complete [ 3877.365032] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3877.382409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3877.395658] Lustre: Skipped 1 previous similar message [ 3877.421689] Lustre: Skipped 1 previous similar message [ 3880.247357] Lustre: server umount lustre-MDT0000 complete [ 3882.465513] LustreError: 121021:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3882.499068] LustreError: 121021:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3886.305512] LustreError: 121002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787342241 with bad export cookie 12092048181047687788 [ 3886.308280] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3886.320248] LustreError: 121002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3886.855344] Lustre: server umount lustre-MDT0001 complete [ 3902.372431] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 3906.835873] Key type lgssc unregistered [ 3907.415432] LNet: 125766:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3907.430183] LNetError: 125766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3907.474417] LNet: Removed LNI 192.168.202.139@tcp [ 3909.059669] Key type .llcrypt unregistered [ 3909.061894] Key type ._llcrypt unregistered [ 3946.763949] Key type ._llcrypt registered [ 3946.766886] Key type .llcrypt registered [ 3946.885752] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 3948.217504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3948.245158] alg: No test for adler32 (adler32-zlib) [ 3949.280377] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [ 3949.524847] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 3951.207307] Key type lgssc registered [ 3952.845966] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3961.891284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3961.907463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3963.400907] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3963.436089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3963.453687] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.39/24@tcp [ 3969.285121] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3973.604186] LustreError: 127332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3973.641384] LustreError: 127332:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3978.732138] LustreError: 127333:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3981.316693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3981.513787] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3981.540562] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.39/24@tcp [ 3981.583533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3985.903530] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3995.241414] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4003.625163] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4012.857982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4013.712615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4021.883808] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 4032.630305] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4033.151460] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 4046.215940] Lustre: server umount lustre-OST0000 complete [ 4053.472240] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4053.490596] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4053.517423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4059.106551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4059.121728] Lustre: Skipped 2 previous similar messages [ 4063.136986] Lustre: 126644:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787342402/real 1787342402] req@00000000aafbb44b x1874164243791360/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787342418 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4063.174102] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4063.184955] Lustre: Skipped 1 previous similar message [ 4063.206993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4063.219072] Lustre: Skipped 1 previous similar message [ 4065.247277] Lustre: 126642:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787342404/real 1787342404] req@00000000b83c0011 x1874164243792128/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787342420 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4065.247990] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4065.557623] Lustre: server umount lustre-MDT0000 complete [ 4068.322540] LustreError: 129952:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4068.357772] LustreError: 129952:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4069.711480] LustreError: 127312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787342424 with bad export cookie 14552745183867494702 [ 4069.731316] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4070.003797] Lustre: server umount lustre-MDT0001 complete [ 4084.257196] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 4088.032859] Key type lgssc unregistered [ 4088.293903] LNet: 130930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4088.300253] LNetError: 130930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4088.326737] LNet: Removed LNI 192.168.202.139@tcp [ 4089.239866] Key type .llcrypt unregistered [ 4089.242743] Key type ._llcrypt unregistered [ 4120.391713] Key type ._llcrypt registered [ 4120.396573] Key type .llcrypt registered [ 4120.547573] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 4139.610443] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 4141.357968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4141.647736] alg: No test for adler32 (adler32-zlib) [ 4143.388279] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [ 4143.810141] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 4145.648858] Key type lgssc registered [ 4148.232105] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4198.036629] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4198.125819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4199.563828] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4199.589302] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4199.634497] Lustre: lustre-MDT0000: new disk, initializing [ 4199.684975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4199.706190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4206.270110] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 4221.191806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4221.371085] Lustre: 134343:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4221.469835] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4221.475870] Lustre: Skipped 1 previous similar message [ 4221.568534] Lustre: lustre-MDT0001: new disk, initializing [ 4221.698144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4221.741468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4221.752813] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4227.943227] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 4239.854721] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4246.464530] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4254.627897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4254.919428] Lustre: lustre-OST0000: new disk, initializing [ 4254.922669] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4254.927684] Lustre: 136311:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4255.029168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4262.052446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4262.070928] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4262.204289] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4262.338540] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 4273.254745] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4278.082743] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4278.262686] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4282.869565] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4283.055741] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4287.972865] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4287.973250] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4287.995974] LustreError: Skipped 1 previous similar message [ 4287.998575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4288.042799] Lustre: Skipped 1 previous similar message [ 4291.295499] Lustre: server umount lustre-OST0000 complete [ 4298.209304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4298.212494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4298.226615] Lustre: Skipped 1 previous similar message [ 4298.241745] Lustre: Skipped 2 previous similar messages [ 4301.531543] Lustre: server umount lustre-MDT0000 complete [ 4306.173618] LustreError: 134334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787342661 with bad export cookie 6250240859538018465 [ 4306.174256] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4306.181348] LustreError: 134334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4306.540726] Lustre: server umount lustre-MDT0001 complete [ 4315.873991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4316.933616] LustreError: 138530:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4316.963202] LustreError: 138530:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4317.162490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4321.771572] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 4322.279868] LustreError: 138531:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4322.297379] LustreError: 138531:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4327.402325] LustreError: 138530:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4331.867583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4332.271151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4337.454479] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 4346.243864] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4353.889658] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4361.502705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4361.948439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4368.941164] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 4377.557863] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4382.838114] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4383.150940] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4389.474344] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4389.767571] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4398.688491] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4398.820494] Lustre: lustre-OST0001: new disk, initializing [ 4398.823394] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4398.828364] Lustre: 141906:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4399.008679] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4407.009380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4407.031353] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4407.260927] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4407.813414] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 4417.720101] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4418.019091] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4423.180239] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4423.414366] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4429.419398] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8abf52e21800.ost_server_uuid 50 [ 4431.277339] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8abf52e21800.ost_server_uuid in FULL state after 0 sec [ 4441.891381] Lustre: server umount lustre-OST0001 complete [ 4443.105534] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4443.107513] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4443.119498] LustreError: Skipped 1 previous similar message [ 4443.140665] LustreError: 140357:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4443.147990] Lustre: Skipped 1 previous similar message [ 4443.199029] LustreError: 140357:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4448.230683] LustreError: 140356:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4448.230928] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4448.234113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4448.243849] LustreError: 140356:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4448.255306] Lustre: Skipped 1 previous similar message [ 4452.464288] Lustre: server umount lustre-OST0000 complete [ 4458.472300] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4458.475788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4458.491729] Lustre: Skipped 1 previous similar message [ 4458.515634] Lustre: Skipped 2 previous similar messages [ 4463.584943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4463.601944] Lustre: Skipped 1 previous similar message [ 4471.263177] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4471.528785] Lustre: server umount lustre-MDT0000 complete [ 4472.823301] LustreError: 138526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4472.847561] LustreError: 138526:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4476.246086] LustreError: 138511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787342831 with bad export cookie 6250240859538019347 [ 4476.254696] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4476.262387] LustreError: 138511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4476.635118] Lustre: server umount lustre-MDT0001 complete [ 4497.851871] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 4500.580551] Key type lgssc unregistered [ 4500.836781] LNet: 144911:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4500.849210] LNetError: 144911:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4500.881345] LNet: Removed LNI 192.168.202.139@tcp [ 4501.706194] Key type .llcrypt unregistered [ 4501.708838] Key type ._llcrypt unregistered [ 4545.121713] Key type ._llcrypt registered [ 4545.125425] Key type .llcrypt registered [ 4545.226808] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 4561.602886] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 4562.848812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4562.941709] alg: No test for adler32 (adler32-zlib) [ 4563.963462] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [ 4564.171513] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 4565.831172] Key type lgssc registered [ 4566.767198] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4605.544557] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4605.564137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4606.887955] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4606.918502] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4606.999634] Lustre: lustre-MDT0000: new disk, initializing [ 4607.077966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4607.106723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4611.761154] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 4622.697717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4622.888243] Lustre: 149067:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4622.987839] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4623.000402] Lustre: Skipped 1 previous similar message [ 4623.109561] Lustre: lustre-MDT0001: new disk, initializing [ 4623.228188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4623.280450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4623.296158] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4628.329939] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 4638.157936] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4644.970445] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4652.503691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4652.938042] Lustre: lustre-OST0000: new disk, initializing [ 4652.941727] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4652.946613] Lustre: 151035:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4653.073886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4659.778933] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 4661.847814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4661.853304] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4661.967155] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4669.325369] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4673.753712] Lustre: Failing over lustre-OST0000 [ 4673.916506] Lustre: server umount lustre-OST0000 complete [ 4692.324906] Lustre: 147291:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787343031/real 1787343031] req@0000000027022a07 x1874164888667136/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787343047 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4692.357534] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4693.407197] Lustre: 147290:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787343032/real 1787343032] req@0000000013e50c4a x1874164888667264/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787343048 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4693.443474] Lustre: 147290:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4693.459797] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4694.943164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4695.252964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4695.286372] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4698.272055] Lustre: 147289:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787343037/real 1787343037] req@00000000b551e2ea x1874164888669056/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787343053 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4698.302089] Lustre: 147289:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4701.607523] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 4702.381055] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4703.497683] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4703.501146] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4704.544509] Lustre: 147290:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787343043/real 1787343043] req@000000000d8d7e99 x1874164888670336/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787343059 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4705.978823] Lustre: Failing over lustre-MDT0000 [ 4706.320807] Lustre: server umount lustre-MDT0000 complete [ 4707.495987] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4707.505886] Lustre: Skipped 2 previous similar messages [ 4712.679779] LustreError: 151495:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4712.712847] LustreError: 151495:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4717.796431] LustreError: 149072:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4717.822429] LustreError: 149072:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4722.907563] LustreError: 151495:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4722.926225] LustreError: 151495:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4723.180089] Lustre: 147291:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787343062/real 1787343062] req@000000003038c982 x1874164888682752/t0(0) o400->MGC192.168.202.139@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787343078 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4723.225379] Lustre: 147291:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4723.247963] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4725.356677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4728.024519] LustreError: 149074:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4728.047227] LustreError: 149074:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4733.410685] LustreError: 147287:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000057a2660e x1874164888687616/t0(0) o250->MGC192.168.202.139@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4733.460064] LustreError: 149077:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4733.490360] LustreError: 149077:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4733.948175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4734.073731] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4734.170559] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4739.056595] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4739.078388] Lustre: Skipped 1 previous similar message [ 4739.083988] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4739.177056] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4739.928479] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 4749.281872] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4749.284571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4749.289190] Lustre: Skipped 1 previous similar message [ 4754.409993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4754.417061] Lustre: Skipped 2 previous similar messages [ 4755.496952] Lustre: server umount lustre-OST0000 complete [ 4759.524118] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4759.528628] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4759.544623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4759.544959] Lustre: Skipped 1 previous similar message [ 4764.642542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4764.649828] Lustre: Skipped 2 previous similar messages [ 4769.761034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4769.764259] Lustre: Skipped 1 previous similar message [ 4773.343628] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4773.577237] Lustre: server umount lustre-MDT0000 complete [ 4774.882025] LustreError: 149077:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4774.908957] LustreError: 149077:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4777.614486] LustreError: 149058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787343132 with bad export cookie 8272199040558615770 [ 4777.629051] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4777.635159] LustreError: 149058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4778.024307] Lustre: server umount lustre-MDT0001 complete [ 4789.846910] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 4792.249477] Key type lgssc unregistered [ 4792.576639] LNet: 154694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4792.581428] LNetError: 154694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4792.592995] LNet: Removed LNI 192.168.202.139@tcp [ 4793.355488] Key type .llcrypt unregistered [ 4793.361938] Key type ._llcrypt unregistered [ 4831.421087] Key type ._llcrypt registered [ 4831.427596] Key type .llcrypt registered [ 4831.564547] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 4833.014037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4833.068700] alg: No test for adler32 (adler32-zlib) [ 4834.209966] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [ 4834.477351] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 4836.167195] Key type lgssc registered [ 4837.323968] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4845.456968] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4845.467709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4846.898199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4852.192960] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 4857.328344] LustreError: 156215:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4857.351059] LustreError: 156215:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4861.917423] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4862.433161] LustreError: 156216:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4865.803297] LustreError: 156244:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4865.812262] LustreError: 156244:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4865.819413] LustreError: 156244:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 4866.198905] Lustre: server umount lustre-MDT0000 complete [ 4883.273234] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 4894.495964] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 4933.120524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4933.324474] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4933.381629] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4933.479167] Lustre: lustre-MDT0000: new disk, initializing [ 4933.559788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4933.583943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4938.198242] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 4950.109433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4950.298583] Lustre: 159563:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4950.381674] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4950.385096] Lustre: Skipped 1 previous similar message [ 4950.502769] Lustre: lustre-MDT0001: new disk, initializing [ 4950.660379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4950.701121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4950.712288] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4956.105283] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 4966.105706] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4973.463103] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4981.536130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4981.953303] Lustre: lustre-OST0000: new disk, initializing [ 4981.957177] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4981.979708] Lustre: 161533:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4982.115850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4989.073177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4989.092570] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4989.287744] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4990.798761] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 5004.433924] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5008.660238] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5010.870521] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 16:16:04 (1787343364) [ 5265.890795] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5265.893685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5265.901360] Lustre: Skipped 2 previous similar messages [ 5265.912349] Lustre: Skipped 2 previous similar messages [ 5271.008924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5271.014780] Lustre: Skipped 2 previous similar messages [ 5276.129084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5276.143694] Lustre: Skipped 2 previous similar messages [ 5278.175363] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5278.368668] Lustre: server umount lustre-MDT0000 complete [ 5281.262099] LustreError: 159574:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5281.290205] LustreError: 159574:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5282.611709] LustreError: 159555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787343637 with bad export cookie 10063317471410010993 [ 5282.613107] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5282.619464] LustreError: 159555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5286.367972] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5286.371475] LustreError: 161746:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5286.401959] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5289.554928] Lustre: server umount lustre-MDT0001 complete [ 5300.609340] Lustre: server umount lustre-OST0000 complete [ 5313.122814] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 5324.833171] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 5365.363886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5365.724834] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5365.802912] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5365.859653] Lustre: lustre-MDT0000: new disk, initializing [ 5365.914741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5365.935393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5371.667857] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 5383.165635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5383.392623] Lustre: 166202:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5383.507313] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5383.513654] Lustre: Skipped 1 previous similar message [ 5383.674593] Lustre: lustre-MDT0001: new disk, initializing [ 5383.773805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5383.832631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5383.847947] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5389.674208] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 5402.348542] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5410.690465] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5418.590634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5419.002556] Lustre: lustre-OST0000: new disk, initializing [ 5419.012129] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5419.022398] Lustre: 168169:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5419.185892] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5424.763958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5424.776300] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5424.898704] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5426.576753] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 5435.995982] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5440.354715] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5440.517348] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5444.270117] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5444.469635] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5450.721094] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5450.724090] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5450.731909] LustreError: Skipped 1 previous similar message [ 5450.738879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5450.738893] Lustre: Skipped 1 previous similar message [ 5450.786487] Lustre: Skipped 1 previous similar message [ 5452.671303] Lustre: server umount lustre-OST0000 complete [ 5460.449686] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5460.461058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5460.475065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5460.482656] Lustre: Skipped 1 previous similar message [ 5463.925845] Lustre: server umount lustre-MDT0000 complete [ 5466.081202] LustreError: 166213:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5466.126236] LustreError: 166213:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5468.654384] LustreError: 168179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787343823 with bad export cookie 10063317471410169830 [ 5468.655918] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5468.671327] LustreError: 168179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5469.277577] Lustre: server umount lustre-MDT0001 complete [ 5479.694766] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 16:23:54 (1787343834) [ 5494.569132] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 5505.137409] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 5540.641044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5541.020928] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5541.066716] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5541.190857] Lustre: lustre-MDT0000: new disk, initializing [ 5541.278105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5541.297615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5545.797266] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 5556.532233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5556.664374] Lustre: 172923:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5557.123829] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5563.290639] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 5572.526684] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5578.213722] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5585.366673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5585.761401] Lustre: lustre-OST0000: new disk, initializing [ 5585.765494] Lustre: Skipped 1 previous similar message [ 5585.771426] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5585.778369] Lustre: Skipped 2 previous similar messages [ 5585.786606] Lustre: 174902:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5592.513404] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 5594.157971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5594.174888] Lustre: Skipped 1 previous similar message [ 5594.182292] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5594.293472] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5604.285528] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5615.072233] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5615.076113] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5615.080132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5615.088936] Lustre: Skipped 2 previous similar messages [ 5615.092651] Lustre: Skipped 1 previous similar message [ 5620.249990] Lustre: server umount lustre-OST0000 complete [ 5623.778885] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5623.794177] LustreError: Skipped 1 previous similar message [ 5623.802676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5638.111877] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5638.750381] Lustre: server umount lustre-MDT0000 complete [ 5640.677046] LustreError: 172936:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5640.688368] LustreError: 172936:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5642.192094] LustreError: 174913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787343997 with bad export cookie 10063317471410170712 [ 5642.204338] LustreError: 174913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5642.209844] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5642.573854] Lustre: server umount lustre-MDT0001 complete [ 5654.990525] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 16:26:48 (1787344008) [ 5672.925664] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 5684.651657] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 5722.073533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5722.400244] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5722.433672] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5722.518300] Lustre: lustre-MDT0000: new disk, initializing [ 5722.611208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5722.616765] Lustre: Skipped 2 previous similar messages [ 5722.647365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5727.025551] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 5737.619566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5737.745322] Lustre: 179231:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5738.008342] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5741.965903] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 5751.171827] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5757.026314] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5763.834779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5764.083648] Lustre: 181207:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5770.817417] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 5772.890952] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5773.000687] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5780.097976] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5788.644136] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5788.651024] Lustre: Skipped 1 previous similar message [ 5788.660342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5788.668374] Lustre: Skipped 7 previous similar messages [ 5788.668467] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5793.293490] Lustre: server umount lustre-OST0000 complete [ 5811.167264] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5811.343363] Lustre: server umount lustre-MDT0000 complete [ 5814.195612] LustreError: 180031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787344169 with bad export cookie 10063317471410171797 [ 5814.201217] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5814.209543] LustreError: 180031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5814.568829] Lustre: server umount lustre-MDT0001 complete [ 5824.330309] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 5826.866564] Key type lgssc unregistered [ 5827.194528] LNet: 183066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5827.197918] LNetError: 183066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5827.210031] LNet: Removed LNI 192.168.202.139@tcp [ 5828.076155] Key type .llcrypt unregistered [ 5828.078377] Key type ._llcrypt unregistered [ 5857.914944] Key type ._llcrypt registered [ 5857.921840] Key type .llcrypt registered [ 5858.066907] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 5858.984247] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5859.028774] alg: No test for adler32 (adler32-zlib) [ 5860.211302] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [ 5860.621882] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 5862.343359] Key type lgssc registered [ 5863.437836] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5871.914200] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5871.931429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5873.547306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5878.833881] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 5882.852711] LustreError: 184492:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5882.893785] LustreError: 184492:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5887.970013] LustreError: 184491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5888.153816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5888.409340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5894.117868] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 5903.122816] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5911.156664] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5919.251718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5919.635726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5926.136155] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 5933.999116] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5943.223889] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5963.752592] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5963.767546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5963.779875] Lustre: Skipped 1 previous similar message [ 5963.787221] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5963.794747] Lustre: Skipped 1 previous similar message [ 5968.187927] Lustre: server umount lustre-OST0000 complete [ 5973.986085] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5973.998617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5974.002425] Lustre: Skipped 1 previous similar message [ 5974.019306] Lustre: Skipped 1 previous similar message [ 5979.109466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5979.597399] Lustre: server umount lustre-MDT0000 complete [ 5982.995569] LustreError: 184470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787344338 with bad export cookie 77078045496875287 [ 5983.000734] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5983.018338] LustreError: 184470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5983.518704] Lustre: server umount lustre-MDT0001 complete [ 5996.228890] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 6000.162367] Key type lgssc unregistered [ 6000.533841] LNet: 188052:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6000.552582] LNetError: 188052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6000.576225] LNet: Removed LNI 192.168.202.139@tcp [ 6001.874522] Key type .llcrypt unregistered [ 6001.877636] Key type ._llcrypt unregistered [ 6029.752874] Key type ._llcrypt registered [ 6029.755400] Key type .llcrypt registered [ 6029.887669] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 6031.830110] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6031.861425] alg: No test for adler32 (adler32-zlib) [ 6032.881466] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [ 6033.092628] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 6034.751891] Key type lgssc registered [ 6036.023228] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6045.337529] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6045.355406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6046.737451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6051.758774] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6055.928894] LustreError: 189471:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6055.959379] LustreError: 189471:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6060.838914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6061.037907] LustreError: 189470:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6061.483038] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6068.398803] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6078.404117] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6085.925865] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6093.550521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6093.967555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6101.663532] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6110.207881] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6116.543316] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6126.316682] Lustre: lustre-OST0000: Not available for connect from 192.168.202.39@tcp (stopping) [ 6127.073574] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6127.076168] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6127.087121] LustreError: Skipped 1 previous similar message [ 6127.088758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6127.110212] Lustre: Skipped 1 previous similar message [ 6127.139972] Lustre: Skipped 1 previous similar message [ 6131.414398] Lustre: lustre-OST0000: Not available for connect from 192.168.202.39@tcp (stopping) [ 6132.214405] LustreError: 191292:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6132.263217] LustreError: 191292:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6132.424962] Lustre: server umount lustre-OST0000 complete [ 6142.953819] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8abf4377e800.ost_server_uuid 50 [ 6145.407650] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6161.889431] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6161.906081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6161.914590] Lustre: Skipped 1 previous similar message [ 6161.929222] Lustre: Skipped 1 previous similar message [ 6167.008765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6167.027459] Lustre: Skipped 1 previous similar message [ 6167.559909] Lustre: server umount lustre-MDT0000 complete [ 6171.781348] LustreError: 189452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787344527 with bad export cookie 1069368455918361059 [ 6171.814552] LustreError: 189452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6171.815101] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6172.200161] Lustre: server umount lustre-MDT0001 complete [ 6185.331672] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 16:35:38 (1787344538) [ 6194.890351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6195.325916] LustreError: 193410:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6195.405935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6200.073944] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6200.803720] LustreError: 193411:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6209.145140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6209.649924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6214.070773] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6221.803886] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6227.564979] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6234.134469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6234.645169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6240.396639] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6240.768454] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6248.656329] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6256.352376] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6256.563793] Lustre: lustre-OST0001: new disk, initializing [ 6256.567376] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6256.572516] Lustre: 196255:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6256.677203] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6262.842985] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6263.965684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6263.975459] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6264.044713] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6272.832893] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6276.581337] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6276.590265] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6276.607204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6279.680720] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6282.154077] Lustre: server umount lustre-OST0000 complete [ 6283.993478] LustreError: 195233:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6284.022052] LustreError: 195233:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6290.279302] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6290.510356] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6294.951844] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6295.219633] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN [ 6300.138762] LustreError: 195615:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6300.167781] LustreError: 195615:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 6310.373191] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6310.376414] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6310.386246] LustreError: Skipped 1 previous similar message [ 6310.398352] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6310.405840] Lustre: Skipped 4 previous similar messages [ 6316.316853] Lustre: server umount lustre-OST0001 complete [ 6322.153150] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6322.163783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6322.189228] Lustre: Skipped 1 previous similar message [ 6327.337565] Lustre: server umount lustre-MDT0000 complete [ 6332.296031] LustreError: 193392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787344687 with bad export cookie 1069368455918362130 [ 6332.302226] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6332.317043] LustreError: 193392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6332.986102] Lustre: server umount lustre-MDT0001 complete [ 6357.547390] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 16:38:31 (1787344711) [ 6366.640876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6367.074586] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6367.170457] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6367.456451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6372.881316] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6382.945534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6383.048275] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6383.119583] Lustre: 199698:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6388.426393] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6397.087773] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6403.360283] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6410.968505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6411.234651] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6418.596694] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6421.748986] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6427.028440] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6433.988496] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6434.075271] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6434.343494] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6434.354264] Lustre: Skipped 2 previous similar messages [ 6440.786810] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6442.248493] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6449.862501] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6452.707662] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6452.722262] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6452.735408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6452.753192] Lustre: Skipped 2 previous similar messages [ 6452.769210] Lustre: Skipped 6 previous similar messages [ 6458.655817] Lustre: server umount lustre-OST0000 complete [ 6461.173731] LustreError: 201548:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6461.196593] LustreError: 201548:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 6478.309979] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6478.314254] LustreError: Skipped 1 previous similar message [ 6481.680798] Lustre: server umount lustre-OST0001 complete [ 6488.545842] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6488.575104] Lustre: Skipped 2 previous similar messages [ 6493.644551] Lustre: server umount lustre-MDT0000 complete [ 6498.275702] LustreError: 199689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787344853 with bad export cookie 1069368455918363537 [ 6498.283208] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6498.296133] LustreError: 199689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6498.804550] Lustre: server umount lustre-MDT0001 complete [ 6525.574631] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 16:41:19 (1787344879) [ 6543.485353] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 6555.568651] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 6594.448424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6594.735965] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6594.795261] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6594.917565] Lustre: lustre-MDT0000: new disk, initializing [ 6595.063762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6595.109735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6600.714283] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6613.631728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6613.758546] Lustre: 207796:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6613.803492] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6613.806108] Lustre: Skipped 1 previous similar message [ 6613.851454] Lustre: lustre-MDT0001: new disk, initializing [ 6613.956296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6613.968382] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6618.664734] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6629.773469] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6636.361917] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6643.288074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6643.590626] Lustre: lustre-OST0000: new disk, initializing [ 6643.595487] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6643.608707] Lustre: 209764:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6650.220068] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6651.476028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6651.486604] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6651.577380] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6659.760662] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6665.460367] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6665.713671] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6671.833697] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6672.204690] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6677.484523] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6677.487865] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6677.491313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6677.491324] Lustre: Skipped 7 previous similar messages [ 6677.514401] Lustre: Skipped 2 previous similar messages [ 6680.716505] Lustre: server umount lustre-OST0000 complete [ 6691.166179] Lustre: server umount lustre-MDT0000 complete [ 6692.837652] LustreError: 207806:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6692.873812] LustreError: 207806:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 6695.769666] LustreError: 207786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787345051 with bad export cookie 1069368455918365315 [ 6695.777957] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6696.280417] Lustre: server umount lustre-MDT0001 complete [ 6705.030468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6710.067849] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6718.981104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6724.988584] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6734.241310] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6741.717058] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6749.501174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6756.490745] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6766.581234] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6771.974779] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6772.285529] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6778.601830] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6778.858684] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6781.923138] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6781.930718] LustreError: Skipped 2 previous similar messages [ 6795.235555] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6795.407402] Lustre: server umount lustre-OST0000 complete [ 6802.159829] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6807.315668] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6836.114300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6842.004741] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6849.196275] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6850.687060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6850.707927] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6850.707972] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6850.729994] Lustre: Skipped 1 previous similar message [ 6853.816030] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 6861.280904] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6861.282344] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6861.295941] LustreError: Skipped 2 previous similar messages [ 6861.296777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6861.296782] Lustre: Skipped 11 previous similar messages [ 6861.329927] Lustre: Skipped 5 previous similar messages [ 6864.086371] Lustre: server umount lustre-OST0000 complete [ 6875.667571] Lustre: server umount lustre-MDT0000 complete [ 6876.654123] LustreError: 212709:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6876.677746] LustreError: 212709:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6880.328852] LustreError: 211963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787345235 with bad export cookie 1069368455918366204 [ 6880.343599] LustreError: 211963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6880.352417] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6880.861948] Lustre: server umount lustre-MDT0001 complete [ 6891.422417] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 16:47:25 (1787345245) [ 6898.821985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6899.333330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6899.338367] Lustre: Skipped 6 previous similar messages [ 6903.796651] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6913.253356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6918.773300] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6928.139749] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6935.300526] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6943.303798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6950.590579] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6958.229669] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6962.848574] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6963.118215] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6967.076612] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6967.272522] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6974.096970] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6974.255356] Lustre: lustre-OST0001: new disk, initializing [ 6974.258602] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6974.263562] Lustre: 221324:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6980.466729] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6984.303239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6984.320549] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6984.428704] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6990.563429] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6996.532131] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6996.836539] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7001.426618] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7001.697131] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7009.931675] Lustre: server umount lustre-OST0001 complete [ 7016.604025] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7016.779941] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7021.412371] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7021.654886] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7052.521500] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7057.907691] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7057.925233] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7057.934096] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7060.419233] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7066.678328] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7068.985293] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7073.280237] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7073.298647] LustreError: Skipped 1 previous similar message [ 7077.408900] Lustre: server umount lustre-OST0001 complete [ 7100.291436] Lustre: server umount lustre-OST0000 complete [ 7119.327447] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7119.329506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7119.334660] Lustre: Skipped 16 previous similar messages [ 7119.614064] Lustre: server umount lustre-MDT0000 complete [ 7123.248696] LustreError: 217937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787345478 with bad export cookie 1069368455918367170 [ 7123.254082] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7123.262594] LustreError: 217937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7129.690358] Lustre: server umount lustre-MDT0001 complete [ 7154.624849] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 16:51:48 (1787345508) [ 7162.136507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7162.514339] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7162.610380] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7167.033302] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7176.848313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7177.003647] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7177.056376] Lustre: 226575:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7182.401798] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7190.629809] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7197.221350] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7203.615473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7203.787193] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7211.013622] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7220.771074] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7232.217726] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7232.403109] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7240.184714] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7249.099982] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7250.878761] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7256.224761] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7256.449248] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7261.388246] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7261.750432] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7268.339662] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8abf435c4800.ost_server_uuid 50 [ 7270.529905] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8abf435c4800.ost_server_uuid in IDLE state after 0 sec [ 7271.997710] Lustre: Permanently deactivating lustre-OST0001 [ 7272.026728] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7276.824172] Lustre: Permanently reactivating lustre-OST0001 [ 7276.870903] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7276.877845] Lustre: Skipped 2 previous similar messages [ 7282.656254] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7282.679048] Lustre: Skipped 10 previous similar messages [ 7287.564504] Lustre: server umount lustre-OST0001 complete [ 7287.785240] LustreError: 228418:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7287.815139] LustreError: 228418:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [ 7297.724417] Lustre: server umount lustre-OST0000 complete [ 7307.554409] Lustre: server umount lustre-MDT0000 complete [ 7310.731088] LustreError: 229448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787345666 with bad export cookie 1069368455918368458 [ 7310.731717] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7310.748701] LustreError: 229448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7310.981234] Lustre: server umount lustre-MDT0001 complete [ 7335.391894] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 16:54:49 (1787345689) [ 7343.309292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7343.625901] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7343.649553] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7343.657267] Lustre: Skipped 2 previous similar messages [ 7347.998664] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7356.116301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7356.268942] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7356.282205] Lustre: 233269:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7362.214105] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7373.492835] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7381.943524] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7389.598376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7389.903317] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7389.943334] Lustre: Permanently deactivating lustre-OST0000 [ 7389.973284] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7395.389179] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7396.118646] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7403.340365] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7411.723564] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7411.858586] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7412.206927] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7412.223054] Lustre: Skipped 11 previous similar messages [ 7418.553315] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7420.871846] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7430.246853] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7431.796195] Lustre: Permanently reactivating lustre-OST0000 [ 7431.812416] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7431.816946] Lustre: Skipped 2 previous similar messages [ 7441.018736] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7441.021912] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 46 seconds. I think it's dead, and I am evicting it. exp 0000000018456b62, cur 1787345796 deadline 1787345780 last 1787345750 [ 7441.040517] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7441.058905] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7441.061682] Lustre: Skipped 1 previous similar message [ 7441.131888] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7449.588352] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7449.603758] LustreError: Skipped 5 previous similar messages [ 7463.391592] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7463.527552] Lustre: server umount lustre-OST0001 complete [ 7473.909075] Lustre: server umount lustre-OST0000 complete [ 7493.087135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7493.208185] Lustre: server umount lustre-MDT0000 complete [ 7497.320669] LustreError: 233264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787345852 with bad export cookie 1069368455918370117 [ 7497.324728] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7497.328366] LustreError: 233264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7498.015982] Lustre: server umount lustre-MDT0001 complete [ 7511.006160] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 7514.473658] Key type lgssc unregistered [ 7514.926107] LNet: 238162:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7514.932526] LNetError: 238162:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7514.960033] LNet: Removed LNI 192.168.202.139@tcp [ 7516.291618] Key type .llcrypt unregistered [ 7516.295631] Key type ._llcrypt unregistered [ 7539.871903] Key type ._llcrypt registered [ 7539.872894] Key type .llcrypt registered [ 7539.927435] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 7540.972500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7541.175227] alg: No test for adler32 (adler32-zlib) [ 7542.466910] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [ 7543.061250] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 7544.943234] Key type lgssc registered [ 7546.192857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7557.122593] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7557.150305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7558.595837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7563.408391] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7567.845339] LustreError: 239632:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7567.880759] LustreError: 239632:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7571.927380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7572.073786] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7572.094236] Lustre: 239621:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7572.100797] Lustre: Permanently deactivating lustre-MDT0001 [ 7572.109991] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7572.445959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7574.756174] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7574.772639] LustreError: 239663:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7574.806588] LustreError: 239663:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7574.835826] LustreError: 239663:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 1, failed: rc = -108 [ 7577.675385] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7585.729401] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7591.735591] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7598.849971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7599.098952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7603.179405] LustreError: 241475:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7603.204378] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7604.790290] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7608.297127] LustreError: 241736:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7612.960326] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7613.414827] LustreError: 241473:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7617.508023] LustreError: 241474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7617.531532] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7617.539776] LustreError: 241474:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7620.125086] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7620.331689] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7625.741502] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7625.760097] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7626.622891] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7636.180662] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7640.244859] Lustre: Permanently reactivating lustre-MDT0001 [ 7640.274407] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7640.282390] Lustre: Skipped 2 previous similar messages [ 7644.418989] Lustre: lustre-MDT0001: Client 973fb4f5-e0ed-450d-a0cf-baa337818deb (at 192.168.202.39@tcp) reconnecting [ 7652.451468] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7654.185329] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7661.069987] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7661.283022] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7668.088330] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7668.337096] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7672.278635] Lustre: Permanently deactivating lustre-MDT0001 [ 7672.296933] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7672.300716] Lustre: Skipped 2 previous similar messages [ 7679.543824] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7688.245093] Lustre: server umount lustre-MDT0000 complete [ 7690.215394] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7690.222379] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7690.244548] LustreError: 239633:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7692.261752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7692.275759] Lustre: Skipped 2 previous similar messages [ 7692.706107] LustreError: 243179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787346047 with bad export cookie 11431972315088164606 [ 7692.712877] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7692.728036] LustreError: 243179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7693.008877] Lustre: server umount lustre-MDT0001 complete [ 7697.369461] Lustre: server umount lustre-OST0000 complete [ 7714.177971] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 17:01:07 (1787346067) [ 7730.081417] Lustre: server umount lustre-OST0001 complete [ 7736.745464] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 7748.667537] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 7790.015288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7790.282801] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7790.294340] Lustre: Skipped 2 previous similar messages [ 7790.311778] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7790.383507] Lustre: lustre-MDT0000: new disk, initializing [ 7790.451503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7790.464254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7796.141187] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7808.600875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7808.757120] Lustre: 247903:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7808.817966] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7808.827641] Lustre: Skipped 1 previous similar message [ 7808.924532] Lustre: lustre-MDT0001: new disk, initializing [ 7809.022441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7809.065310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7809.078218] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7814.255757] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7826.557169] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7834.408091] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7843.858946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7844.247866] Lustre: lustre-OST0000: new disk, initializing [ 7844.252405] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7844.263589] Lustre: 249872:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7844.389371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7851.596715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7851.617868] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7851.759770] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7851.953663] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7862.668231] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7865.635580] LustreError: 247910:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7875.641061] LustreError: 247910:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7875.692547] LustreError: 249508:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7875.749757] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7876.020740] Lustre: lustre-OST0001: new disk, initializing [ 7876.029500] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7876.040695] Lustre: 250989:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7883.334584] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7885.407577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7885.433326] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7885.594701] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7885.750525] LustreError: 249508:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7895.411698] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7897.269292] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7901.170591] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7901.178169] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7901.181116] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7901.190528] Lustre: Skipped 1 previous similar message [ 7901.200971] Lustre: Skipped 1 previous similar message [ 7906.146576] Lustre: server umount lustre-OST0001 complete [ 7906.282643] LustreError: 249855:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7906.322207] LustreError: 249855:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7926.756347] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7926.761282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7926.769750] Lustre: Skipped 1 previous similar message [ 7926.778875] Lustre: Skipped 1 previous similar message [ 7930.621282] Lustre: server umount lustre-OST0000 complete [ 7936.993531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7937.004133] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7937.004371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7937.013214] Lustre: Skipped 1 previous similar message [ 7937.025286] LustreError: Skipped 1 previous similar message [ 7941.960161] Lustre: server umount lustre-MDT0000 complete [ 7942.119680] LustreError: 249271:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7942.139440] LustreError: 249271:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 7946.868601] LustreError: 247894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787346302 with bad export cookie 11431972315088167427 [ 7946.870186] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7946.882722] LustreError: 247894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7953.361790] Lustre: server umount lustre-MDT0001 complete [ 7966.966539] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 7970.164562] Key type lgssc unregistered [ 7970.441731] LNet: 253027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7970.447598] LNetError: 253027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7970.459301] LNet: Removed LNI 192.168.202.139@tcp [ 7971.449771] Key type .llcrypt unregistered [ 7971.451589] Key type ._llcrypt unregistered [ 8016.377182] Key type ._llcrypt registered [ 8016.378955] Key type .llcrypt registered [ 8016.530427] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 8017.749556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8017.767570] alg: No test for adler32 (adler32-zlib) [ 8019.432937] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [ 8019.873150] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 8021.735769] Key type lgssc registered [ 8023.616867] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8034.813383] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8034.851584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8036.189056] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8036.241580] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8036.545662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8042.289927] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8053.548463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8053.715569] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8053.756950] Lustre: 255235:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8053.983315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8060.189897] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8071.513510] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8080.060478] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8088.426989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8088.725670] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8089.180646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8095.528284] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8097.771422] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8109.342429] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8167.392682] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8167.409547] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8167.416463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8167.444481] Lustre: Skipped 1 previous similar message [ 8167.463816] Lustre: Skipped 1 previous similar message [ 8171.121168] Lustre: server umount lustre-OST0000 complete [ 8176.785636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8193.425811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8193.968750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8201.733404] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8207.287831] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8207.308566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8207.317983] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8207.338756] Lustre: Skipped 1 previous similar message [ 8212.060328] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8258.538502] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8258.549816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8258.552128] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8258.552136] LustreError: Skipped 1 previous similar message [ 8258.556778] Lustre: Skipped 1 previous similar message [ 8258.630926] Lustre: Skipped 1 previous similar message [ 8261.910045] Lustre: server umount lustre-OST0000 complete [ 8268.780037] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8268.784776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8268.806509] Lustre: Skipped 1 previous similar message [ 8268.821724] Lustre: Skipped 1 previous similar message [ 8273.679523] Lustre: server umount lustre-MDT0000 complete [ 8277.912623] LustreError: 255227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787346633 with bad export cookie 12249024616829304883 [ 8277.937323] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8278.572141] Lustre: server umount lustre-MDT0001 complete [ 8293.794347] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 8297.198650] Key type lgssc unregistered [ 8297.506208] LNet: 260669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8297.517637] LNetError: 260669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8298.600701] LNet: Removed LNI 192.168.202.139@tcp [ 8299.864660] Key type .llcrypt unregistered [ 8299.867908] Key type ._llcrypt unregistered [ 8332.177291] Key type ._llcrypt registered [ 8332.180251] Key type .llcrypt registered [ 8332.293163] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 8333.685200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8333.845936] alg: No test for adler32 (adler32-zlib) [ 8334.867311] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [ 8335.104378] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 8336.775598] Key type lgssc registered [ 8337.789393] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8345.303982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8345.317434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8346.909481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8351.989655] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8357.346273] LustreError: 262089:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8357.374445] LustreError: 262089:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8362.465395] LustreError: 262090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8363.092596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8363.528101] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8370.249249] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8380.528818] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8388.373745] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8397.463244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8397.836304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8406.176751] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8416.452786] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8417.406250] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8448.490757] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8448.497769] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8448.529107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8453.460217] Lustre: server umount lustre-OST0000 complete [ 8458.725379] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8458.739728] Lustre: Skipped 2 previous similar messages [ 8458.739971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8458.761052] Lustre: Skipped 2 previous similar messages [ 8463.840694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8463.854795] Lustre: Skipped 1 previous similar message [ 8464.371929] Lustre: server umount lustre-MDT0000 complete [ 8467.939956] LustreError: 262090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8467.968403] LustreError: 262090:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8468.765178] LustreError: 262070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787346824 with bad export cookie 16460648141705791017 [ 8468.776976] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8469.286764] Lustre: server umount lustre-MDT0001 complete [ 8484.830193] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 8489.068662] Key type lgssc unregistered [ 8489.490602] LNet: 266398:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8489.497690] LNetError: 266398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8490.534657] LNet: Removed LNI 192.168.202.139@tcp [ 8491.845913] Key type .llcrypt unregistered [ 8491.853551] Key type ._llcrypt unregistered [ 8509.306759] Key type ._llcrypt registered [ 8509.308986] Key type .llcrypt registered [ 8509.409754] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 8510.376750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8510.448907] alg: No test for adler32 (adler32-zlib) [ 8511.546979] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [ 8511.819616] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 8513.488340] Key type lgssc registered [ 8514.548559] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8521.839544] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8521.876156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8523.383291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8528.645463] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8532.962868] LustreError: 267622:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8533.001716] LustreError: 267622:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8538.081136] LustreError: 267623:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8540.550953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8541.433478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8546.170406] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8554.719637] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8561.555400] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8570.132824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8570.513444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8578.002915] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8585.903179] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8593.077151] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8608.749092] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8608.750965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8608.752752] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8608.768253] Lustre: Skipped 1 previous similar message [ 8608.783033] Lustre: Skipped 1 previous similar message [ 8609.227813] Lustre: server umount lustre-OST0000 complete [ 8613.860292] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8613.868955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8613.885867] Lustre: Skipped 1 previous similar message [ 8613.919182] Lustre: Skipped 1 previous similar message [ 8618.979533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8618.988682] Lustre: Skipped 1 previous similar message [ 8620.177429] Lustre: server umount lustre-MDT0000 complete [ 8624.098655] LustreError: 268792:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8624.122717] LustreError: 268792:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8624.677694] LustreError: 267602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787346979 with bad export cookie 12210236766880987169 [ 8624.682091] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8624.688526] LustreError: 267602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8625.082718] Lustre: server umount lustre-MDT0001 complete [ 8635.679732] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 8638.646152] Key type lgssc unregistered [ 8638.990669] LNet: 271324:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8638.997562] LNetError: 271324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8640.048347] LNet: Removed LNI 192.168.202.139@tcp [ 8641.225088] Key type .llcrypt unregistered [ 8641.228435] Key type ._llcrypt unregistered [ 8665.979553] Key type ._llcrypt registered [ 8665.982349] Key type .llcrypt registered [ 8666.060742] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 8666.938438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8667.190559] alg: No test for adler32 (adler32-zlib) [ 8668.437571] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [ 8668.833204] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 8670.591811] Key type lgssc registered [ 8671.928209] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8683.328419] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8683.424603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8684.832523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8690.276336] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8695.266921] LustreError: 272743:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8695.318988] LustreError: 272743:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8700.389391] LustreError: 272744:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8700.449220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8700.890586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8705.879239] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8714.337967] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8722.237935] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8730.652786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8731.082633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8738.827967] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8747.977623] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8755.691337] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8786.401216] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8786.406097] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8786.436877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8786.445084] Lustre: Skipped 1 previous similar message [ 8790.725940] Lustre: server umount lustre-OST0000 complete [ 8796.641773] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8796.648544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8796.657053] Lustre: Skipped 2 previous similar messages [ 8796.673711] Lustre: Skipped 1 previous similar message [ 8801.767372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8801.781274] Lustre: Skipped 1 previous similar message [ 8802.062271] Lustre: server umount lustre-MDT0000 complete [ 8805.859955] LustreError: 272744:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8805.880853] LustreError: 272744:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8806.404103] LustreError: 272723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787347161 with bad export cookie 9292863410632958351 [ 8806.423946] LustreError: 272723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8806.448636] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8806.886967] Lustre: server umount lustre-MDT0001 complete [ 8818.810636] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 8822.593907] Key type lgssc unregistered [ 8823.028923] LNet: 277076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8823.049107] LNetError: 277076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8823.075432] LNet: Removed LNI 192.168.202.139@tcp [ 8824.476128] Key type .llcrypt unregistered [ 8824.479063] Key type ._llcrypt unregistered [ 8845.508569] Key type ._llcrypt registered [ 8845.511909] Key type .llcrypt registered [ 8845.638950] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 8846.684690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8846.873389] alg: No test for adler32 (adler32-zlib) [ 8847.856825] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [ 8848.060535] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 8849.711285] Key type lgssc registered [ 8851.908307] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8860.832969] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8860.853505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8862.626908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8867.605763] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8871.910181] LustreError: 278319:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8871.919580] LustreError: 278319:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8877.032546] LustreError: 278321:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8878.325443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8878.858528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8884.748274] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8895.309373] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8901.912359] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8910.211541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8910.611072] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8920.671513] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8932.190149] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8935.073640] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8952.803369] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8952.815739] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8952.824663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8952.859062] Lustre: Skipped 1 previous similar message [ 8952.883808] Lustre: Skipped 1 previous similar message [ 8957.539786] Lustre: server umount lustre-OST0000 complete [ 8963.040503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8963.087287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8963.097031] Lustre: Skipped 1 previous similar message [ 8968.160827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8968.175565] Lustre: Skipped 1 previous similar message [ 8973.282291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8973.289064] Lustre: Skipped 1 previous similar message [ 8976.351148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8976.614573] Lustre: server umount lustre-MDT0000 complete [ 8978.405320] LustreError: 278327:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8978.437650] LustreError: 278327:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8980.909669] LustreError: 278286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787347336 with bad export cookie 10003832191358880908 [ 8980.911738] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8980.926041] LustreError: 278286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8981.403591] Lustre: server umount lustre-MDT0001 complete [ 8997.463787] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 9002.373658] Key type lgssc unregistered [ 9002.928334] LNet: 282031:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9002.949048] LNetError: 282031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9004.013834] LNet: Removed LNI 192.168.202.139@tcp [ 9005.447586] Key type .llcrypt unregistered [ 9005.449981] Key type ._llcrypt unregistered [ 9021.525299] Key type ._llcrypt registered [ 9021.528795] Key type .llcrypt registered [ 9021.687785] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9045.756334] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 9069.138428] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 9070.141702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9070.375810] alg: No test for adler32 (adler32-zlib) [ 9071.681388] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [ 9071.910557] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 9073.647368] Key type lgssc registered [ 9075.661588] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9123.529332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9123.540570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9124.919902] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9124.948733] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9125.075195] Lustre: lustre-MDT0000: new disk, initializing [ 9125.145623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9125.164636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9130.553096] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9143.169476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9143.417783] Lustre: 285588:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9143.580120] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9143.588581] Lustre: Skipped 1 previous similar message [ 9143.734455] Lustre: lustre-MDT0001: new disk, initializing [ 9143.957044] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9144.013511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9144.026567] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9149.188639] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9162.004154] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9169.266112] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9177.067813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9177.500873] Lustre: lustre-OST0000: new disk, initializing [ 9177.505557] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9177.532385] Lustre: 287559:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9177.659886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9183.317332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9183.336865] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9183.477543] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9185.218645] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9197.696436] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9204.507515] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9204.738776] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9211.632182] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9212.035238] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9219.553735] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9219.559572] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9219.568195] LustreError: Skipped 1 previous similar message [ 9219.571886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9219.587394] Lustre: Skipped 1 previous similar message [ 9221.365860] Lustre: server umount lustre-OST0000 complete [ 9229.804336] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9229.826364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9229.840213] Lustre: Skipped 1 previous similar message [ 9229.851244] Lustre: Skipped 2 previous similar messages [ 9233.851968] Lustre: server umount lustre-MDT0000 complete [ 9238.991288] LustreError: 285580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787347594 with bad export cookie 4885562807155152776 [ 9239.014322] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9239.018372] LustreError: 285580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9239.697786] Lustre: server umount lustre-MDT0001 complete [ 9250.738050] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 17:26:45 (1787347605) [ 9258.599768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9259.238977] LustreError: 289977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9259.281112] LustreError: 289977:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9259.348248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9264.620148] LustreError: 289978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9265.682968] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9268.769771] LustreError: 289977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9273.833966] LustreError: 289978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9278.949682] LustreError: 289977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9279.940696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9280.829444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9290.067042] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9302.844565] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9311.087975] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9319.001568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9319.252936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9326.071168] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9335.292796] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9350.116252] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9350.151048] Lustre: Skipped 1 previous similar message [ 9350.153177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9350.157330] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9350.172261] Lustre: Skipped 1 previous similar message [ 9355.240283] LustreError: 291801:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9355.265717] LustreError: 291801:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9355.622794] Lustre: server umount lustre-OST0000 complete [ 9362.917051] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9362.932871] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9362.952249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9368.174102] Lustre: server umount lustre-MDT0000 complete [ 9372.372082] LustreError: 289957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787347727 with bad export cookie 4885562807155153658 [ 9372.391939] LustreError: 289957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9372.401251] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9373.062433] Lustre: server umount lustre-MDT0001 complete [ 9388.746690] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 9392.887721] Key type lgssc unregistered [ 9393.387583] LNet: 293541:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9393.402699] LNetError: 293541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9393.421232] LNet: Removed LNI 192.168.202.139@tcp [ 9394.920655] Key type .llcrypt unregistered [ 9394.923889] Key type ._llcrypt unregistered [ 9428.597715] Key type ._llcrypt registered [ 9428.599320] Key type .llcrypt registered [ 9428.795445] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 9448.550296] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 9450.101496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9450.343209] alg: No test for adler32 (adler32-zlib) [ 9451.531147] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [ 9451.794238] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 9453.527194] Key type lgssc registered [ 9455.079830] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9517.174566] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9517.213151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9518.440534] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9518.474531] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9518.564675] Lustre: lustre-MDT0000: new disk, initializing [ 9518.634409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9518.653628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9524.175660] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9537.058132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9537.165334] Lustre: 297372:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9537.208851] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9537.211214] Lustre: Skipped 1 previous similar message [ 9537.269908] Lustre: lustre-MDT0001: new disk, initializing [ 9537.355196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9537.386103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9537.395598] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9543.228168] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9554.876444] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9562.521747] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9570.869170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9571.248867] Lustre: lustre-OST2710: new disk, initializing [ 9571.252987] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9571.260523] Lustre: 299341:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9571.375549] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9579.315519] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9580.164374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9580.195931] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9580.392722] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9591.317318] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9600.942875] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9601.265441] Lustre: lustre-OST03e8: new disk, initializing [ 9601.267476] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9601.271687] Lustre: 300407:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9601.343262] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9609.405233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9609.415836] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9609.605912] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9610.075429] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9621.388498] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9631.058995] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9631.359480] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9639.134876] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9639.407545] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9646.414443] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9646.740896] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9653.849741] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9654.293929] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9665.510103] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9665.527465] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9665.545048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9666.036509] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9666.057322] Lustre: Skipped 1 previous similar message [ 9666.083057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9666.092593] Lustre: Skipped 3 previous similar messages [ 9667.839430] Lustre: server umount lustre-MDT0000 complete [ 9672.848756] LustreError: 300406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787348028 with bad export cookie 17749378480268053704 [ 9672.852247] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9672.860268] LustreError: 300406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9673.581924] Lustre: server umount lustre-MDT0001 complete [ 9689.042570] Lustre: server umount lustre-OST2710 complete [ 9691.762136] Lustre: 295171:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787348031/real 1787348031] req@00000000d944ba5b x1874170014154112/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787348047 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9691.818578] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9691.842168] Lustre: Skipped 1 previous similar message [ 9693.666361] Lustre: 295169:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787348032/real 1787348032] req@0000000091b0025c x1874170014154368/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787348048 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9695.665775] Lustre: server umount lustre-OST03e8 complete [ 9718.008825] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 9732.619825] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 9783.290291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9783.770499] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9783.843917] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9783.970621] Lustre: lustre-MDT0000: new disk, initializing [ 9784.171324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9784.224357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9791.145184] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9803.015603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9803.199348] Lustre: 305676:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9803.265662] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9803.274351] Lustre: Skipped 1 previous similar message [ 9803.384860] Lustre: lustre-MDT0001: new disk, initializing [ 9803.505732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9803.541461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9803.563577] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9809.169409] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9824.112868] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9832.169833] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9843.093333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9843.646140] Lustre: lustre-OST0000: new disk, initializing [ 9843.658139] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9843.679586] Lustre: 307649:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9843.950524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9849.554068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9849.580099] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9849.877951] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9853.970911] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9865.258590] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9871.760278] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9872.053616] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9877.004891] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9877.244770] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9880.549462] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9880.553985] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9880.584726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9880.597032] Lustre: Skipped 1 previous similar message [ 9880.633924] Lustre: Skipped 1 previous similar message [ 9885.666105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9885.680288] Lustre: Skipped 1 previous similar message [ 9890.788715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9890.801045] Lustre: Skipped 1 previous similar message [ 9893.344020] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9893.578290] Lustre: server umount lustre-OST0000 complete [ 9901.028253] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9901.031071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9901.044018] LustreError: Skipped 1 previous similar message [ 9901.094528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9906.096917] Lustre: server umount lustre-MDT0000 complete [ 9906.144430] LustreError: 308044:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9906.160103] LustreError: 308044:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9911.131139] LustreError: 305670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787348266 with bad export cookie 17749378480268055391 [ 9911.131643] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9911.145869] LustreError: 305670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9912.231474] Lustre: server umount lustre-MDT0001 complete [ 9922.378879] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 17:37:56 (1787348276) [ 9924.653932] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9927.061644] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 17:38:01 (1787348281) [ 9932.132139] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 9968.528349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9968.897434] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9968.953120] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9969.252843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9974.678981] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9984.623816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9984.740207] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9984.773604] Lustre: 311512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9993.026620] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10018.680101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10018.979111] Lustre: Denying initial registration attempt from nid 192.168.202.139@tcp, specified as failover [10018.989077] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [10019.007881] LustreError: 312966:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [10019.026574] LustreError: 312966:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10019.049885] LustreError: 312966:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10019.136604] Lustre: server umount lustre-OST0000 complete [10019.139663] LustreError: 312966:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [10027.489331] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10027.494086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10027.503259] Lustre: Skipped 2 previous similar messages [10027.521776] Lustre: Skipped 2 previous similar messages [10041.313370] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10041.705792] Lustre: server umount lustre-MDT0000 complete [10042.849088] LustreError: 311524:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10042.866437] LustreError: 311524:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10046.691308] LustreError: 311505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787348401 with bad export cookie 17749378480268056280 [10046.708729] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10046.713540] LustreError: 311505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10047.011899] Lustre: server umount lustre-MDT0001 complete [10063.804778] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [10067.071609] Key type lgssc unregistered [10067.270603] LNet: 314116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10067.276611] LNetError: 314116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10067.289914] LNet: Removed LNI 192.168.202.139@tcp [10067.896784] Key type .llcrypt unregistered [10067.899535] Key type ._llcrypt unregistered [10082.103268] Key type ._llcrypt registered [10082.105308] Key type .llcrypt registered [10082.197258] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [10082.946066] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10082.984763] alg: No test for adler32 (adler32-zlib) [10084.114926] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [10084.305688] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [10085.943392] Key type lgssc registered [10087.847314] Lustre: Echo OBD driver; http://www.lustre.org/ [10146.502602] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [10157.580881] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10157.606832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10158.928854] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10158.947616] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10159.281568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10165.454348] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10175.203898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10175.286201] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10175.311903] Lustre: 316886:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10175.528061] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10180.405992] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10190.037725] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10195.984558] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10216.695697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10217.007137] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10217.305237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10224.054876] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10232.259843] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10235.872646] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10235.884822] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10235.898810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10236.388341] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10236.421528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10236.427340] Lustre: Skipped 1 previous similar message [10240.802512] Lustre: server umount lustre-OST0000 complete [10246.629943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10246.638718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10246.653795] Lustre: Skipped 1 previous similar message [10251.751943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10251.776604] Lustre: Skipped 2 previous similar messages [10253.002341] Lustre: server umount lustre-MDT0000 complete [10260.077499] LustreError: 316878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787348615 with bad export cookie 16109451781252552924 [10260.080478] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10260.089877] LustreError: 316878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10261.044259] Lustre: server umount lustre-MDT0001 complete [10274.720793] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [10277.716385] Key type lgssc unregistered [10278.083635] LNet: 320469:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10278.097294] LNetError: 320469:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10278.112282] LNet: Removed LNI 192.168.202.139@tcp [10278.967157] Key type .llcrypt unregistered [10278.968951] Key type ._llcrypt unregistered [10314.510111] Key type ._llcrypt registered [10314.511421] Key type .llcrypt registered [10314.738937] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [10315.600308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10315.665203] alg: No test for adler32 (adler32-zlib) [10316.622427] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [10316.904956] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [10318.536461] Key type lgssc registered [10321.083317] Lustre: Echo OBD driver; http://www.lustre.org/ [10334.444563] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10334.468436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10336.299024] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10342.002641] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10346.472901] LustreError: 321887:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10346.484177] LustreError: 321887:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10351.585151] LustreError: 321886:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10352.772688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10353.307876] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10359.964966] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10373.015305] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10380.465175] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10391.361910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10391.924778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10398.461829] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10408.546721] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10419.681122] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10419.702963] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10419.724890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10420.510049] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.39@tcp (stopping) [10421.732059] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10421.733055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10421.766403] Lustre: Skipped 1 previous similar message [10421.793650] Lustre: Skipped 2 previous similar messages [10424.676082] Lustre: server umount lustre-MDT0000 complete [10425.557209] LustreError: 321882:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10429.991791] LustreError: 321868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787348785 with bad export cookie 1794059493832409280 [10430.017683] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10430.679551] LustreError: 321883:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10430.704947] LustreError: 321883:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10430.721401] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.39@tcp (stopping) [10431.968520] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10434.976916] LustreError: 321882:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10434.979300] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10435.008671] LustreError: 321882:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10435.048491] Lustre: Skipped 1 previous similar message [10436.535491] Lustre: server umount lustre-MDT0001 complete [10444.717860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10457.440773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10474.990997] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x18e5c79ab3cc5cab [10475.007689] Lustre: MGC192.168.202.139@tcp: Connection restored to 0@lo (at 0@lo) [10475.506242] LustreError: 323720:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10475.536679] LustreError: 323720:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10475.644941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10475.783357] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10481.128297] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10490.360687] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10495.105806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10495.192704] LustreError: 325484:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10495.222421] LustreError: 325484:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10495.471702] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10495.477781] Lustre: Skipped 1 previous similar message [10495.709657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10501.104864] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10501.209135] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10510.122553] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10512.487903] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10519.162910] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10521.317204] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10531.809183] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10531.815719] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10531.832097] LustreError: Skipped 1 previous similar message [10531.833072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10531.899737] Lustre: Skipped 1 previous similar message [10533.413859] Lustre: server umount lustre-OST0000 complete [10541.540871] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10541.543737] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10544.674068] Lustre: server umount lustre-MDT0000 complete [10547.170079] LustreError: 325489:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10547.229139] LustreError: 325489:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10550.882753] LustreError: 321867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787348906 with bad export cookie 1794059493832416427 [10550.897493] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10550.914552] LustreError: 321867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [10551.410985] Lustre: server umount lustre-MDT0001 complete [10569.985964] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [10575.257293] Key type lgssc unregistered [10575.814819] LNet: 328164:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10575.838412] LNetError: 328164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10575.880455] LNet: Removed LNI 192.168.202.139@tcp [10577.538764] Key type .llcrypt unregistered [10577.549287] Key type ._llcrypt unregistered [10608.788255] Key type ._llcrypt registered [10608.790890] Key type .llcrypt registered [10608.899365] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [10611.021996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10611.089889] alg: No test for adler32 (adler32-zlib) [10612.141497] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [10612.461239] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [10614.151160] Key type lgssc registered [10615.793316] Lustre: Echo OBD driver; http://www.lustre.org/ [10625.605679] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10625.650343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10627.100342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10631.867236] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10637.284052] LustreError: 329583:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10637.313210] LustreError: 329583:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10641.831813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10642.478318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10647.618937] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10656.248244] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10662.712304] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10667.107630] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10667.114763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10667.120257] Lustre: Skipped 1 previous similar message [10672.608407] LustreError: 330311:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10672.641940] LustreError: 330311:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10672.643144] Lustre: server umount lustre-MDT0000 complete [10676.364405] LustreError: 329565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787349031 with bad export cookie 1028182941464639154 [10676.375525] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10677.061297] Lustre: server umount lustre-MDT0001 complete [10694.654220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10694.929589] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10694.981383] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10695.281626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10699.678986] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10709.195029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10709.351555] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10709.408305] Lustre: 332169:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10709.671319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10714.452479] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10722.988477] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10728.951233] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10739.503571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10739.867694] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10739.878914] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10739.887161] LustreError: 334053:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10739.900385] LustreError: 334053:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10739.909479] LustreError: 334053:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10739.997197] Lustre: server umount lustre-OST0000 complete [10739.999829] LustreError: 334053:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10753.715899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10754.172355] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10754.583294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10761.370616] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10762.191231] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10770.204882] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10780.662373] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10780.829929] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10780.959313] Lustre: lustre-OST0001: new disk, initializing [10780.962349] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10780.976925] Lustre: 335600:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10781.087641] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10789.996958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10790.014849] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10790.177079] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10791.854554] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10803.577992] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10810.863191] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10810.882270] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10810.905415] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10810.926037] Lustre: Skipped 1 previous similar message [10810.985271] Lustre: Skipped 2 previous similar messages [10812.943329] Lustre: server umount lustre-OST0001 complete [10816.015696] LustreError: 334522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10816.024262] LustreError: 334522:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10819.046305] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10819.068076] LustreError: 334521:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10819.071981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10819.075245] Lustre: Skipped 1 previous similar message [10819.096053] LustreError: 334521:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10824.163324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10824.168471] LustreError: 334840:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10824.178469] Lustre: Skipped 2 previous similar messages [10824.190410] LustreError: 334840:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10829.307164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10829.326055] Lustre: Skipped 1 previous similar message [10833.377604] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10833.931433] Lustre: server umount lustre-OST0000 complete [10843.105906] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10843.133155] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10843.178539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10846.952254] Lustre: server umount lustre-MDT0000 complete [10849.762272] LustreError: 333719:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10849.798096] LustreError: 333719:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10853.530888] LustreError: 332163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787349208 with bad export cookie 1028182941464639707 [10853.551689] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10853.561934] LustreError: 332163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10854.158409] Lustre: server umount lustre-MDT0001 complete [10873.352527] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [10879.799138] Key type lgssc unregistered [10880.205963] LNet: 337580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10880.219262] LNetError: 337580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10881.258443] LNet: Removed LNI 192.168.202.139@tcp [10882.374055] Key type .llcrypt unregistered [10882.378222] Key type ._llcrypt unregistered [10961.345467] Key type ._llcrypt registered [10961.348395] Key type .llcrypt registered [10961.506709] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [10977.178856] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [10978.473142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10978.560365] alg: No test for adler32 (adler32-zlib) [10979.671584] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [10979.947174] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [10981.631191] Key type lgssc registered [10982.744666] Lustre: Echo OBD driver; http://www.lustre.org/ [11021.887532] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11021.929199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11023.557827] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11023.607217] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11024.018155] Lustre: lustre-MDT0000: new disk, initializing [11024.219106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11024.279279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11031.813668] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11047.624394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11047.873042] Lustre: 342514:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11047.967979] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11047.979698] Lustre: Skipped 1 previous similar message [11048.175721] Lustre: lustre-MDT0001: new disk, initializing [11048.292501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11048.337378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11048.357076] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11052.467899] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11062.904500] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11069.048612] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11076.941206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11077.357138] Lustre: lustre-OST0000: new disk, initializing [11077.365738] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11077.369610] Lustre: 344480:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11077.467068] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11083.831383] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11085.942931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11085.961973] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11086.169556] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11094.034697] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11099.751390] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11100.064412] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11106.612616] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11106.815611] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11111.914492] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11111.922750] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11111.929847] Lustre: Skipped 1 previous similar message [11111.933095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11111.964931] LustreError: Skipped 1 previous similar message [11116.311443] Lustre: server umount lustre-OST0000 complete [11122.145742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11122.157386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11122.168564] Lustre: Skipped 2 previous similar messages [11127.220782] Lustre: server umount lustre-MDT0000 complete [11131.418795] LustreError: 342504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787349486 with bad export cookie 15678567713958896775 [11131.421763] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11131.425453] LustreError: 342504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11131.953387] Lustre: server umount lustre-MDT0001 complete [11142.003591] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 17:58:16 (1787349496) [11150.224523] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 17:58:24 (1787349504) [11159.149573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11159.571582] LustreError: 347241:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11159.595252] LustreError: 347241:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11159.644209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11164.158730] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11164.657474] LustreError: 347242:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11168.736539] LustreError: 347241:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11172.424133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11172.914601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11177.252692] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11185.869345] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11192.359692] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11199.678583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11200.181538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11208.006449] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11216.440510] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11228.530376] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11230.993575] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11232.826609] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11234.991480] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11237.411677] Lustre: Failing over lustre-MDT0000 [11237.878676] Lustre: server umount lustre-MDT0000 complete [11238.652673] LustreError: 347237:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11239.400040] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11239.404108] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11239.409702] Lustre: Skipped 2 previous similar messages [11243.737278] LustreError: 347237:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11243.769459] LustreError: 347237:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11253.975039] LustreError: 348779:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11254.018792] LustreError: 348779:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [11254.751369] Lustre: 340738:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787349594/real 1787349594] req@00000000e11f0f70 x1874171616372608/t0(0) o400->MGC192.168.202.139@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787349610 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11254.785966] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11257.494445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11265.347221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11265.417719] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11270.069816] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11270.381430] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11270.641063] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11270.726142] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [11270.789471] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11280.377566] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11282.310920] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11286.059319] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11288.301962] Lustre: Failing over lustre-MDT0000 [11288.708893] Lustre: server umount lustre-MDT0000 complete [11290.887845] LustreError: 347237:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11290.908282] LustreError: 347237:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [11291.105424] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11291.133482] Lustre: Skipped 2 previous similar messages [11307.487518] Lustre: 340738:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787349646/real 1787349646] req@000000005c11b4b1 x1874171616393216/t0(0) o400->MGC192.168.202.139@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787349662 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11307.533630] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11308.193528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11317.743739] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xd9957644696daf23 [11317.760165] Lustre: MGC192.168.202.139@tcp: Connection restored to 0@lo (at 0@lo) [11317.768058] Lustre: Skipped 2 previous similar messages [11318.212692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11318.282389] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11321.575360] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11323.189114] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11323.377160] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11323.426480] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [11323.473177] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11333.722676] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11335.662511] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11343.841802] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11343.845265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11343.867594] Lustre: Skipped 3 previous similar messages [11343.882423] Lustre: Skipped 2 previous similar messages [11348.007343] Lustre: server umount lustre-MDT0000 complete [11348.963366] LustreError: 348779:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11348.971071] LustreError: 348779:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 28 previous similar messages [11351.859844] LustreError: 347223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787349707 with bad export cookie 15678567713958899491 [11351.861192] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11351.879140] LustreError: 347223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11352.344697] Lustre: server umount lustre-MDT0001 complete [11366.112680] Lustre: server umount lustre-OST0000 complete [11382.344178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11382.902556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11387.084757] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11397.552763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11405.287000] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11416.465750] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11422.974468] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11431.052416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11437.546490] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11445.548891] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11452.977399] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11463.144609] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11463.154428] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11463.176071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11468.263171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11468.279340] Lustre: Skipped 2 previous similar messages [11469.001226] Lustre: server umount lustre-OST0000 complete [11478.496901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11478.507748] Lustre: Skipped 2 previous similar messages [11479.603411] Lustre: server umount lustre-MDT0000 complete [11483.619064] LustreError: 353604:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11483.639704] LustreError: 353604:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11485.109574] LustreError: 353591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787349840 with bad export cookie 15678567713958900233 [11485.114887] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11485.128022] LustreError: 353591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11485.500196] Lustre: server umount lustre-MDT0001 complete [11499.380974] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [11502.125575] Key type lgssc unregistered [11502.357873] LNet: 357171:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11502.373294] LNetError: 357171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11502.387440] LNet: Removed LNI 192.168.202.139@tcp [11503.179698] Key type .llcrypt unregistered [11503.181481] Key type ._llcrypt unregistered [11531.051293] Key type ._llcrypt registered [11531.054847] Key type .llcrypt registered [11531.164721] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [11532.166445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11532.246052] alg: No test for adler32 (adler32-zlib) [11533.306628] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [11533.494933] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [11535.136764] Key type lgssc registered [11536.213576] Lustre: Echo OBD driver; http://www.lustre.org/ [11544.053633] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11544.074307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11545.583128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11550.593584] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11554.799971] LustreError: 358691:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11554.817414] LustreError: 358691:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11558.706624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11559.005379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11563.274706] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11572.221620] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11580.989694] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11591.865393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11592.356162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11601.223593] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11610.490923] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11614.989876] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11630.577969] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11630.591274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11630.595958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11630.600879] Lustre: Skipped 2 previous similar messages [11634.929658] Lustre: server umount lustre-MDT0000 complete [11635.684319] LustreError: 358692:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11635.719739] LustreError: 358692:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11640.199606] LustreError: 358671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787349995 with bad export cookie 6005615616773062314 [11640.255777] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11640.804119] Lustre: server umount lustre-MDT0001 complete [11655.229651] Lustre: server umount lustre-OST0000 complete [11671.900989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11672.276359] LustreError: 362423:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11672.387903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11677.666301] LustreError: 362424:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11678.764146] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11681.786832] LustreError: 362423:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11689.154549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11689.416374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11694.868862] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11705.573453] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11714.537321] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11722.009519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11722.251082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11729.578415] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11738.913156] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11741.763216] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11761.125162] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11761.137791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11761.159534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11761.172887] Lustre: Skipped 2 previous similar messages [11762.149050] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11762.170420] Lustre: Skipped 1 previous similar message [11762.195963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11762.204309] Lustre: Skipped 2 previous similar messages [11764.514311] Lustre: server umount lustre-MDT0000 complete [11767.287382] LustreError: 362424:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11767.325852] LustreError: 362424:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11769.981033] LustreError: 362404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787350125 with bad export cookie 6005615616773063427 [11769.983897] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11770.000801] LustreError: 362404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [11770.687745] Lustre: server umount lustre-MDT0001 complete [11787.960661] Lustre: server umount lustre-OST0000 complete [11808.230188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11808.972772] LustreError: 366339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11809.172628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11815.810156] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11827.533308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11832.647968] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11843.004235] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11849.690955] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11857.607232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11858.143229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11858.160711] Lustre: Skipped 1 previous similar message [11868.059402] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11878.854362] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11879.631303] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11889.133417] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11889.150780] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11889.168971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11889.189459] Lustre: Skipped 1 previous similar message [11891.620922] Lustre: server umount lustre-OST0000 complete [11899.363542] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11899.385500] Lustre: Skipped 1 previous similar message [11899.395692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11902.253873] Lustre: server umount lustre-MDT0000 complete [11904.486960] LustreError: 366340:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11904.496523] LustreError: 366340:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11907.048405] LustreError: 366320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787350262 with bad export cookie 6005615616773064449 [11907.069681] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11907.628599] Lustre: server umount lustre-MDT0001 complete [11920.365065] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [11923.008786] Key type lgssc unregistered [11923.334585] LNet: 369879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11923.346449] LNetError: 369879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11923.366598] LNet: Removed LNI 192.168.202.139@tcp [11924.298550] Key type .llcrypt unregistered [11924.300497] Key type ._llcrypt unregistered [11955.329258] Key type ._llcrypt registered [11955.334316] Key type .llcrypt registered [11955.485853] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [11956.681400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11956.804097] alg: No test for adler32 (adler32-zlib) [11957.961673] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [11958.343292] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [11959.999169] Key type lgssc registered [11961.763654] Lustre: Echo OBD driver; http://www.lustre.org/ [11970.949712] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11970.963396] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11970.974452] LustreError: 371322:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11971.039921] LustreError: 371322:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11971.050156] LustreError: 371322:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11971.058971] LustreError: 371322:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11971.068147] LustreError: 371322:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11980.060268] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11980.065793] LustreError: 371719:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11980.081738] LustreError: 371719:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11980.087775] LustreError: 371719:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11980.094921] LustreError: 371719:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11980.102599] LustreError: 371719:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [12000.130955] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [12002.698685] Key type lgssc unregistered [12003.016830] LNet: 372726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12003.023958] LNetError: 372726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12003.041872] LNet: Removed LNI 192.168.202.139@tcp [12003.869756] Key type .llcrypt unregistered [12003.874124] Key type ._llcrypt unregistered [12026.304880] Key type ._llcrypt registered [12026.310328] Key type .llcrypt registered [12026.497945] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [12044.478690] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [12045.756130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12045.957090] alg: No test for adler32 (adler32-zlib) [12047.150361] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [12047.488287] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [12049.255845] Key type lgssc registered [12050.391086] Lustre: Echo OBD driver; http://www.lustre.org/ [12091.790821] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12091.837626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12093.216035] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12093.292890] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12093.394789] Lustre: lustre-MDT0000: new disk, initializing [12093.495115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12093.527543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12098.773935] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [12110.924773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12111.111699] Lustre: 375883:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12111.199712] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12111.209513] Lustre: Skipped 1 previous similar message [12111.346852] Lustre: lustre-MDT0001: new disk, initializing [12111.486189] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12111.522645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12111.543626] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12116.370249] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [12127.277352] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12133.065750] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12141.190067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12141.651909] Lustre: lustre-OST0000: new disk, initializing [12141.655912] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12141.661896] Lustre: 377855:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12141.749903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12149.014243] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [12150.434608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12150.446560] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12150.578910] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12160.472076] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12166.774893] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12167.058703] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12172.826146] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12173.020927] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12176.353381] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12176.355894] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12176.377597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12176.386019] Lustre: Skipped 1 previous similar message [12181.484632] Lustre: server umount lustre-OST0000 complete [12186.594647] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12186.595322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12186.632638] Lustre: Skipped 2 previous similar messages [12191.720940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12191.731901] Lustre: Skipped 2 previous similar messages [12192.901923] Lustre: server umount lustre-MDT0000 complete [12198.547051] LustreError: 375875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787350553 with bad export cookie 10951930017912129410 [12198.557939] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12198.564169] LustreError: 375875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12199.234967] Lustre: server umount lustre-MDT0001 complete [12212.489643] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [12214.488643] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 18:16:08 (1787350568) [12222.232246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12222.627498] LustreError: 380316:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12222.679927] LustreError: 380316:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12222.750403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12228.077839] LustreError: 380317:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12228.854371] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [12232.178713] LustreError: 380316:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12237.294740] LustreError: 380317:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12239.018507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12239.544263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12243.909203] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [12253.660181] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12260.490817] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12269.347561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12270.342931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12277.682250] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [12286.738426] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12295.721075] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12296.038791] Lustre: lustre-OST0001: new disk, initializing [12296.042514] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12296.049689] Lustre: 383155:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12296.204447] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12302.468358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12302.480577] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12302.679072] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12304.135765] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [12315.925230] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12323.093518] Lustre: lustre-OST0001: Not available for connect from 192.168.202.39@tcp (stopping) [12323.299734] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12323.304823] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12323.309827] Lustre: Skipped 1 previous similar message [12323.322888] LustreError: Skipped 1 previous similar message [12326.416615] Lustre: server umount lustre-OST0001 complete [12328.158964] LustreError: 382971:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12338.661266] LustreError: 382140:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12338.675182] LustreError: 382140:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12346.337399] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12346.356628] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12346.396359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12346.404314] Lustre: Skipped 2 previous similar messages [12348.389638] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12350.177678] Lustre: server umount lustre-OST0000 complete [12357.094411] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12357.125938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12357.171568] Lustre: Skipped 1 previous similar message [12357.194131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12357.199235] Lustre: Skipped 2 previous similar messages [12360.586106] Lustre: server umount lustre-MDT0000 complete [12364.256940] LustreError: 382160:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12364.286863] LustreError: 382160:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12364.750750] LustreError: 380296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787350720 with bad export cookie 10951930017912130292 [12364.761206] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12364.765288] LustreError: 380296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12379.103430] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12379.418918] Lustre: server umount lustre-MDT0001 complete [12393.132834] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [12397.223707] Key type lgssc unregistered [12397.707154] LNet: 385147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12397.714307] LNetError: 385147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12398.757983] LNet: Removed LNI 192.168.202.139@tcp [12399.942927] Key type .llcrypt unregistered [12399.945486] Key type ._llcrypt unregistered [12460.341491] Key type ._llcrypt registered [12460.343339] Key type .llcrypt registered [12460.414719] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [12461.995715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12462.570426] alg: No test for adler32 (adler32-zlib) [12463.880257] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [12464.329293] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [12466.063638] Key type lgssc registered [12468.163048] Lustre: Echo OBD driver; http://www.lustre.org/ [12478.876160] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12478.918160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12480.397336] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12480.437181] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12480.627426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12485.654369] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [12498.224034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12498.449126] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12498.482892] Lustre: 387947:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12498.920268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12504.302608] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [12514.960993] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12523.168863] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12525.537926] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12525.557018] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12525.561912] Lustre: Skipped 1 previous similar message [12530.663030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12530.672914] Lustre: Skipped 1 previous similar message [12531.994337] Lustre: server umount lustre-MDT0000 complete [12536.822863] LustreError: 387940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787350892 with bad export cookie 15175334341139479975 [12536.828190] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12536.837821] LustreError: 387940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12537.292285] Lustre: server umount lustre-MDT0001 complete [12549.174829] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 18:21:43 (1787350903) [12558.217790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12558.817249] LustreError: 390465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12558.827825] LustreError: 390465:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12558.879629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12563.948169] LustreError: 390466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12564.987767] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [12569.066455] LustreError: 390465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12574.197477] LustreError: 390466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12576.243831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12577.005455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12583.296473] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [12595.595355] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12603.166740] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12612.933239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12613.344407] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12613.746216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12621.514866] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [12632.201517] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12641.682443] Lustre: Permanently deactivating lustre-OST0000 [12641.698321] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12650.583874] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12652.278121] LustreError: 393455:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12652.302742] LustreError: 393455:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12652.314397] LustreError: 393455:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12657.398839] LustreError: 393504:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12657.407894] LustreError: 393504:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12662.651152] Lustre: server umount lustre-OST0000 complete [12666.914380] LustreError: 393754:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12666.921287] LustreError: 393754:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12669.425468] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12669.433340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12669.446078] Lustre: Skipped 1 previous similar message [12669.455373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12673.506500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12673.507149] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12678.624190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12678.629686] Lustre: Skipped 2 previous similar messages [12679.901180] Lustre: server umount lustre-MDT0000 complete [12683.747145] LustreError: 391192:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12683.775802] LustreError: 391192:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12684.567957] LustreError: 390447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787351039 with bad export cookie 15175334341139480556 [12684.575268] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12684.587274] LustreError: 390447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12694.265956] Lustre: server umount lustre-MDT0001 complete [12706.213231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12711.397465] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [12714.461557] LustreError: 394913:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12722.620717] Lustre: 395253:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.139@tcp [12725.332500] Lustre: server umount MGS complete [12739.889845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12740.387051] LustreError: 395978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12740.417528] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12740.422609] Lustre: Skipped 1 previous similar message [12740.482154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12746.369707] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [12758.883500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12759.202250] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12759.295545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12765.091523] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [12777.034524] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12787.390637] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12790.226873] Lustre: Permanently reactivating lustre-OST0000 [12790.262288] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12790.270070] Lustre: Skipped 2 previous similar messages [12816.865513] LustreError: 397714:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12828.721188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12829.400644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12840.490682] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [12851.042202] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12858.336847] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12858.336930] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12858.339975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12858.430789] Lustre: Skipped 1 previous similar message [12858.906746] Lustre: server umount lustre-OST0000 complete [12863.457419] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12863.493188] Lustre: Skipped 2 previous similar messages [12869.816275] Lustre: server umount lustre-MDT0000 complete [12873.703794] LustreError: 395978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12873.720859] LustreError: 395978:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12874.237366] LustreError: 395958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787351229 with bad export cookie 15175334341139481599 [12874.246490] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12874.255444] LustreError: 395958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12874.794498] Lustre: server umount lustre-MDT0001 complete [12891.762285] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [12894.978463] Key type lgssc unregistered [12895.324036] LNet: 400254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12895.329628] LNetError: 400254:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12895.341994] LNet: Removed LNI 192.168.202.139@tcp [12896.575324] Key type .llcrypt unregistered [12896.585439] Key type ._llcrypt unregistered [12919.784645] Key type ._llcrypt registered [12919.792351] Key type .llcrypt registered [12919.966536] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [12939.529516] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [12941.086227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12941.199175] alg: No test for adler32 (adler32-zlib) [12942.376734] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [12942.626759] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [12944.343204] Key type lgssc registered [12945.920180] Lustre: Echo OBD driver; http://www.lustre.org/ [12984.678689] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12984.712887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12986.065731] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12986.094789] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12986.204352] Lustre: lustre-MDT0000: new disk, initializing [12986.282246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12986.318174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12991.503559] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [13005.109183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13005.257578] Lustre: 403413:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13005.309822] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13005.313165] Lustre: Skipped 1 previous similar message [13005.379395] Lustre: lustre-MDT0001: new disk, initializing [13005.477571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13005.507900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13005.527137] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13010.201593] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [13021.762384] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13029.459849] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13039.077127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13039.472648] Lustre: lustre-OST0000: new disk, initializing [13039.475930] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13039.480787] Lustre: 405382:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13039.600373] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13046.358133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13046.373982] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13046.501434] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13047.662347] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [13058.865893] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13063.772859] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13064.112614] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13069.507797] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13069.781389] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13072.367062] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13072.369077] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13072.378014] LustreError: Skipped 1 previous similar message [13072.383542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13072.390095] Lustre: Skipped 1 previous similar message [13072.422782] Lustre: Skipped 1 previous similar message [13077.500503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13077.515130] Lustre: Skipped 1 previous similar message [13078.742882] Lustre: server umount lustre-OST0000 complete [13087.713130] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13087.719305] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13087.729572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13087.746318] Lustre: Skipped 1 previous similar message [13090.146609] Lustre: server umount lustre-MDT0000 complete [13095.302540] LustreError: 403405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787351450 with bad export cookie 16582454918235375499 [13095.303768] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13095.326191] LustreError: 403405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13096.094709] Lustre: server umount lustre-MDT0001 complete [13109.293145] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [13111.618845] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 18:31:05 (1787351465) [13121.208135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13121.814658] LustreError: 407845:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13121.825790] LustreError: 407845:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13121.928932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13127.139096] LustreError: 407846:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13128.146422] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [13132.258529] LustreError: 407845:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13137.292582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13137.378112] LustreError: 407846:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13137.742470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13142.887830] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [13151.029429] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13158.128570] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13166.893427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13167.360262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13175.983789] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [13186.408711] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13188.679409] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [13191.602549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [13199.330072] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13199.340425] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13199.357654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13199.368543] Lustre: Skipped 1 previous similar message [13202.401075] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13202.405959] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13204.272833] Lustre: server umount lustre-OST0000 complete [13209.577097] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13209.587629] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13209.606175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13209.609824] Lustre: Skipped 2 previous similar messages [13214.721770] Lustre: server umount lustre-MDT0000 complete [13217.762472] LustreError: 407845:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13217.781631] LustreError: 407845:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13219.303518] LustreError: 407825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787351574 with bad export cookie 16582454918235376381 [13219.321424] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13219.323981] LustreError: 407825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13219.657676] Lustre: server umount lustre-MDT0001 complete [13233.465418] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [13237.162815] Key type lgssc unregistered [13237.599645] LNet: 411576:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13237.608744] LNetError: 411576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13237.632961] LNet: Removed LNI 192.168.202.139@tcp [13238.767088] Key type .llcrypt unregistered [13238.768629] Key type ._llcrypt unregistered [13270.716442] Key type ._llcrypt registered [13270.723839] Key type .llcrypt registered [13270.843970] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [13271.229292] Key type .llcrypt unregistered [13271.233983] Key type ._llcrypt unregistered [13290.259471] Key type ._llcrypt registered [13290.260803] Key type .llcrypt registered [13290.346522] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [13291.607721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13291.621702] alg: No test for adler32 (adler32-zlib) [13292.727723] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [13293.106971] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [13294.799402] Key type lgssc registered [13296.044799] Lustre: Echo OBD driver; http://www.lustre.org/ [13303.733253] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13303.744544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13305.202757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13309.991725] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [13315.555180] LustreError: 413795:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13315.565432] LustreError: 413795:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13320.680221] LustreError: 413796:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13321.206881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13321.499829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13326.819042] LustreError: 413795:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13326.838761] LustreError: 413795:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13328.813807] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [13329.891931] LustreError: 413796:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13329.922521] LustreError: 413796:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13329.947934] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13335.016079] LustreError: 413795:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13335.043395] LustreError: 413795:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13338.185797] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13345.254195] LustreError: 415113:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13345.286169] LustreError: 415113:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13347.986958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13348.592144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13354.299036] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [13364.195363] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13364.201450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13364.225149] Lustre: Skipped 1 previous similar message [13364.225792] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13364.249023] Lustre: Skipped 1 previous similar message [13366.125226] Lustre: server umount lustre-OST0000 complete [13370.365916] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13370.374979] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13370.384405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13370.664537] Lustre: server umount lustre-MDT0000 complete [13374.433122] LustreError: 415905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13374.434346] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13374.702962] LustreError: 413778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787351729 with bad export cookie 18275477871541610160 [13374.703558] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13374.726470] LustreError: 413778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13375.254844] Lustre: server umount lustre-MDT0001 complete [13389.357682] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [13392.536264] Key type lgssc unregistered [13392.921160] LNet: 416952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13392.933288] LNetError: 416952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13393.959745] LNet: Removed LNI 192.168.202.139@tcp [13394.845066] Key type .llcrypt unregistered [13394.847651] Key type ._llcrypt unregistered [13426.896994] Key type ._llcrypt registered [13426.899870] Key type .llcrypt registered [13427.031857] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [13428.202531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13428.379172] alg: No test for adler32 (adler32-zlib) [13429.453302] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [13429.733572] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [13431.423199] Key type lgssc registered [13432.472713] Lustre: Echo OBD driver; http://www.lustre.org/ [13441.585973] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13441.597080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13457.121687] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13462.239971] LustreError: 418345:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.139@tcp: failed processing log, type 4: rc = -110 [13488.928535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13495.507253] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [13504.202819] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13513.165497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13513.810284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13513.876497] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13518.824199] LustreError: 418371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13518.846859] LustreError: 418371:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13519.238842] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [13522.914717] LustreError: 418370:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13522.933618] LustreError: 418370:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13528.063534] LustreError: 419382:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13528.089595] LustreError: 419382:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13528.970557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13529.504753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13534.456236] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [13543.320609] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13549.904690] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13559.825545] Lustre: server umount lustre-OST0000 complete [13564.192149] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13564.205831] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13564.229247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13565.408640] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13565.411676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13565.452251] Lustre: Skipped 1 previous similar message [13570.529770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13574.734064] Lustre: server umount lustre-MDT0000 complete [13575.652273] LustreError: 419387:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13575.674587] LustreError: 419387:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13576.671126] Lustre: 417682:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787351915/real 1787351915] req@00000000c1ae59fe x1874174185338880/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787351931 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13576.683067] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13578.316081] LustreError: 418351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787351933 with bad export cookie 4551871181217393602 [13578.320030] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13578.329989] LustreError: 418351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13578.767720] Lustre: server umount lustre-MDT0001 complete [13592.606168] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [13596.205117] Key type lgssc unregistered [13596.568259] LNet: 421919:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13596.595859] LNetError: 421919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13596.634627] LNet: Removed LNI 192.168.202.139@tcp [13597.537345] Key type .llcrypt unregistered [13597.540390] Key type ._llcrypt unregistered [13629.461477] Key type ._llcrypt registered [13629.464304] Key type .llcrypt registered [13629.555753] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [13631.039850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13631.192418] alg: No test for adler32 (adler32-zlib) [13632.216353] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [13632.566203] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [13634.287228] Key type lgssc registered [13636.054769] Lustre: Echo OBD driver; http://www.lustre.org/ [13644.701123] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13644.718592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13646.301841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13651.765750] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [13656.545913] LustreError: 423338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13656.575272] LustreError: 423338:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13661.669709] LustreError: 423339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13664.049825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13664.685500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13670.224560] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [13680.057185] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13687.308657] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13696.028274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13696.511776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13703.540195] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [13712.459759] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13715.960144] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13724.422644] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.39@tcp (stopping) [13726.178829] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13726.180694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13726.181165] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13726.204598] Lustre: Skipped 2 previous similar messages [13726.215559] Lustre: Skipped 1 previous similar message [13728.424167] Lustre: server umount lustre-MDT0000 complete [13729.504676] LustreError: 423334:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13731.332659] LustreError: 425184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.39@tcp arrived at 1787352086 with bad export cookie 7840846073515406873 [13731.345351] LustreError: 425184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13733.406733] LustreError: 423319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787352088 with bad export cookie 7840846073515406208 [13733.425977] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13733.440978] LustreError: 423319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13734.368628] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13734.380249] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13734.415239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13734.432092] Lustre: Skipped 1 previous similar message [13736.416758] LustreError: 423338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13736.420837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13736.442517] LustreError: 423338:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13739.688210] Lustre: server umount lustre-OST0000 complete [13743.587369] LustreError: 423338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13743.625989] LustreError: 423338:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13747.428485] Lustre: server umount lustre-MDT0001 complete [13759.757452] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [13762.560034] Key type lgssc unregistered [13762.828881] LNet: 427030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13762.839172] LNetError: 427030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13762.852883] LNet: Removed LNI 192.168.202.139@tcp [13763.437247] Key type .llcrypt unregistered [13763.439689] Key type ._llcrypt unregistered [13794.676485] Key type ._llcrypt registered [13794.679987] Key type .llcrypt registered [13794.813882] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [13796.172820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13796.246061] alg: No test for adler32 (adler32-zlib) [13797.353304] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [13797.643064] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [13799.352064] Key type lgssc registered [13800.735318] Lustre: Echo OBD driver; http://www.lustre.org/ [13809.515546] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13809.533273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13811.232223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13816.493165] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [13821.418880] LustreError: 428448:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13821.441722] LustreError: 428448:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13826.330664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13826.542563] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [13826.787097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13831.301655] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [13840.163604] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13848.413831] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13857.059618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13857.338607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13864.077886] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [13872.069788] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13881.899690] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13888.486942] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13888.490624] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13888.502479] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13892.069214] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13892.073409] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13892.081598] Lustre: Skipped 1 previous similar message [13894.365051] LustreError: 428442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13894.400662] Lustre: server umount lustre-MDT0001 complete [13897.186852] LustreError: 430937:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13897.205991] LustreError: 430937:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13902.306683] LustreError: 428448:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13902.321164] LustreError: 428448:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13907.433392] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13907.435038] LustreError: 430937:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13907.442103] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13907.465976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13907.511403] Lustre: Skipped 1 previous similar message [13910.178579] Lustre: server umount lustre-OST0000 complete [13915.253087] Lustre: server umount lustre-MDT0000 complete [13931.400922] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [13934.326487] Key type lgssc unregistered [13934.678810] LNet: 432158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13934.688763] LNetError: 432158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13934.708863] LNet: Removed LNI 192.168.202.139@tcp [13935.707471] Key type .llcrypt unregistered [13935.709625] Key type ._llcrypt unregistered [13965.328951] Key type ._llcrypt registered [13965.335366] Key type .llcrypt registered [13965.462897] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [13982.170513] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [13984.120525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13984.184873] alg: No test for adler32 (adler32-zlib) [13985.543978] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [13985.906658] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [13987.615401] Key type lgssc registered [13989.357429] Lustre: Echo OBD driver; http://www.lustre.org/ [14034.040886] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [14052.643800] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14052.701438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14054.028264] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14054.079971] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14054.229640] Lustre: lustre-MDT0000: new disk, initializing [14054.356269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14054.394838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14059.442979] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [14077.485988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14077.784228] Lustre: lustre-OST0000: new disk, initializing [14077.787786] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14077.791718] Lustre: Skipped 1 previous similar message [14077.796628] Lustre: 437194:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14077.846426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14084.583567] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [14087.741283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14087.762208] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14087.805731] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14094.278855] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14112.705883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14112.864658] Lustre: 436176:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14112.945548] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14113.016279] Lustre: lustre-MDT0001: new disk, initializing [14113.128954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14113.219965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [14113.234063] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [14118.672388] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [14128.750996] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [14128.984708] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [14134.512254] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [14134.898769] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [14142.948225] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14142.952719] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [14142.973512] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14143.741170] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.39@tcp (stopping) [14143.757515] Lustre: Skipped 1 previous similar message [14147.708244] Lustre: server umount lustre-MDT0001 complete [14153.197244] LustreError: 438745:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14153.209198] LustreError: 438745:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [14158.307488] LustreError: 436184:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14158.333495] LustreError: 436184:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14163.436044] LustreError: 436187:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14163.460027] LustreError: 436187:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14168.548995] LustreError: 438907:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14168.550852] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14168.601929] Lustre: Skipped 1 previous similar message [14168.608657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14173.671066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14173.680238] LustreError: 436187:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14178.802214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14181.345047] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14181.572679] Lustre: server umount lustre-OST0000 complete [14183.906339] LustreError: 436187:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14183.923984] LustreError: 436187:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14200.287240] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14200.806821] Lustre: server umount lustre-MDT0000 complete [14217.619909] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [14220.885831] Key type lgssc unregistered [14221.333572] LNet: 440947:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14221.355940] LNetError: 440947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14222.372825] LNet: Removed LNI 192.168.202.139@tcp [14223.373889] Key type .llcrypt unregistered [14223.376213] Key type ._llcrypt unregistered [14269.718259] Key type ._llcrypt registered [14269.720046] Key type .llcrypt registered [14269.822585] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [14270.100518] Key type .llcrypt unregistered [14270.102224] Key type ._llcrypt unregistered [14289.767956] Key type ._llcrypt registered [14289.770558] Key type .llcrypt registered [14289.904078] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [14291.814831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14291.899299] alg: No test for adler32 (adler32-zlib) [14293.046460] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [14293.325302] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [14294.991278] Key type lgssc registered [14296.264391] Lustre: Echo OBD driver; http://www.lustre.org/ [14322.308648] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14322.358532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14323.687047] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14323.743525] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14323.851587] Lustre: lustre-MDT0000: new disk, initializing [14323.966987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14323.989996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14328.794775] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [14339.381886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14339.556319] Lustre: 444335:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14339.652528] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14339.661893] Lustre: Skipped 1 previous similar message [14339.766374] Lustre: lustre-MDT0001: new disk, initializing [14339.886826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14339.954056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14339.970612] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14344.460031] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [14355.388352] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14363.013379] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14371.650036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14371.881861] Lustre: lustre-OST0000: new disk, initializing [14371.884331] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14371.892023] Lustre: 446304:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14372.068465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14379.601134] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [14379.610617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14379.626189] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14379.747637] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14390.611973] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14396.384612] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14396.399181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14396.413720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14400.481410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14400.503016] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14402.481623] Lustre: server umount lustre-MDT0000 complete [14405.985288] LustreError: 446302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787352761 with bad export cookie 15296545826405840963 [14405.998013] LustreError: 446302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14405.998632] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14406.286562] Lustre: server umount lustre-MDT0001 complete [14420.598504] Lustre: server umount lustre-OST0000 complete [14431.103584] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 18:53:05 (1787352785) [14439.016911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14439.571340] LustreError: 448332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14439.589507] LustreError: 448332:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [14439.648887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14444.079719] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [14445.032969] LustreError: 448333:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14449.121753] LustreError: 448332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14452.150470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14452.530823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14456.900811] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [14464.482325] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14471.324959] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14492.016980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14492.215963] Lustre: Found index 0 for lustre-OST0000, updating log [14492.396852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14498.318863] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [14505.711734] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14509.811487] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14518.241951] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14518.252072] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14518.259257] Lustre: Skipped 1 previous similar message [14518.262977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14518.266542] Lustre: Skipped 2 previous similar messages [14522.215921] Lustre: server umount lustre-OST0000 complete [14539.747383] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14539.760368] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14539.777174] Lustre: Skipped 1 previous similar message [14539.781631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14539.785404] Lustre: Skipped 3 previous similar messages [14544.864626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14544.873445] Lustre: Skipped 3 previous similar messages [14545.890686] LustreError: 448332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14545.910705] LustreError: 448332:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14547.197904] Lustre: server umount lustre-MDT0000 complete [14550.990554] LustreError: 448313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787352906 with bad export cookie 15296545826405842356 [14550.996638] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14551.003714] LustreError: 448313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14551.450697] Lustre: server umount lustre-MDT0001 complete [14556.513197] LNet: 1 local NIs in recovery (showing 1): 192.168.202.139@tcp [14560.951293] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 18:55:15 (1787352915) [15184.005664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15184.860832] LustreError: 454753:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15184.986918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15189.993551] LustreError: 454754:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15190.934477] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [15195.106955] LustreError: 454753:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15201.256166] LustreError: 454754:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15201.280531] LustreError: 454754:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15201.608219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15202.474350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15207.691485] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [15216.627724] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15223.673826] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15234.399989] LNet: 1 local NIs in recovery (showing 1): 192.168.202.139@tcp [15243.979124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15244.354638] Lustre: Found index 0 for lustre-OST0000, updating log [15244.636635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15250.672523] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [15258.652820] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15261.939079] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15261.956091] Lustre: Skipped 1 previous similar message [15261.967342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15263.094140] Lustre: server umount lustre-OST0000 complete [15268.832429] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15268.843644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15268.861891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15269.929190] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15271.010606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15271.021899] Lustre: Skipped 3 previous similar messages [15274.779144] Lustre: server umount lustre-MDT0000 complete [15275.124597] LustreError: 454750:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15275.142773] LustreError: 454750:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15279.595440] LustreError: 454733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787353634 with bad export cookie 15296545826405843441 [15279.602942] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15279.622634] LustreError: 454733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15280.113821] Lustre: server umount lustre-MDT0001 complete [15295.841071] LNet: 1 local NIs in recovery (showing 1): 192.168.202.139@tcp [15310.807650] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [15322.062387] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [15363.943818] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 19:08:37 (1787353717) [15386.605452] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [15389.701719] Key type lgssc unregistered [15389.976758] LNet: 461371:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15389.984637] LNetError: 461371:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15504.680509] LNet: Removed LNI 192.168.202.139@tcp [15505.767188] Key type .llcrypt unregistered [15505.769189] Key type ._llcrypt unregistered [15527.174822] Key type ._llcrypt registered [15527.179622] Key type .llcrypt registered [15527.312045] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [15528.794367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15528.878711] alg: No test for adler32 (adler32-zlib) [15529.899302] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [15530.065222] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [15531.711230] Key type lgssc registered [15533.134814] Lustre: Echo OBD driver; http://www.lustre.org/ [15537.192649] LNet: Added LNI 192.168.202.139@tcp100 [8/256/0/180] [15538.071776] LNet: Added LNI 192.168.202.139@tcp101 [8/256/0/180] [15538.953564] LNet: Added LNI 192.168.202.139@tcp102 [8/256/0/180] [15539.826942] LNet: Added LNI 192.168.202.139@tcp103 [8/256/0/180] [15540.684496] LNet: Added LNI 192.168.202.139@tcp104 [8/256/0/180] [15541.483612] LNet: Added LNI 192.168.202.139@tcp105 [8/256/0/180] [15542.255858] LNet: Added LNI 192.168.202.139@tcp106 [8/256/0/180] [15543.128446] LNet: Added LNI 192.168.202.139@tcp107 [8/256/0/180] [15543.963585] LNet: Added LNI 192.168.202.139@tcp108 [8/256/0/180] [15544.728391] LNet: Added LNI 192.168.202.139@tcp109 [8/256/0/180] [15545.711965] LNet: Added LNI 192.168.202.139@tcp110 [8/256/0/180] [15546.622711] LNet: Added LNI 192.168.202.139@tcp111 [8/256/0/180] [15547.459286] LNet: Added LNI 192.168.202.139@tcp112 [8/256/0/180] [15548.242402] LNet: Added LNI 192.168.202.139@tcp113 [8/256/0/180] [15549.047502] LNet: Added LNI 192.168.202.139@tcp114 [8/256/0/180] [15549.965917] LNet: Added LNI 192.168.202.139@tcp115 [8/256/0/180] [15550.827867] LNet: Added LNI 192.168.202.139@tcp116 [8/256/0/180] [15551.608512] LNet: Added LNI 192.168.202.139@tcp117 [8/256/0/180] [15552.462078] LNet: Added LNI 192.168.202.139@tcp118 [8/256/0/180] [15553.418639] LNet: Added LNI 192.168.202.139@tcp119 [8/256/0/180] [15554.306916] LNet: Added LNI 192.168.202.139@tcp120 [8/256/0/180] [15555.096383] LNet: Added LNI 192.168.202.139@tcp121 [8/256/0/180] [15555.942254] LNet: Added LNI 192.168.202.139@tcp122 [8/256/0/180] [15556.847809] LNet: Added LNI 192.168.202.139@tcp123 [8/256/0/180] [15557.765494] LNet: Added LNI 192.168.202.139@tcp124 [8/256/0/180] [15558.779941] LNet: Added LNI 192.168.202.139@tcp125 [8/256/0/180] [15559.915429] LNet: Added LNI 192.168.202.139@tcp126 [8/256/0/180] [15560.829657] LNet: Added LNI 192.168.202.139@tcp127 [8/256/0/180] [15561.888594] LNet: Added LNI 192.168.202.139@tcp128 [8/256/0/180] [15562.919645] LNet: Added LNI 192.168.202.139@tcp129 [8/256/0/180] [15563.961530] LNet: Added LNI 192.168.202.139@tcp130 [8/256/0/180] [15564.653372] LNet: Added LNI 192.168.202.139@tcp131 [8/256/0/180] [15565.776513] LNet: Added LNI 192.168.202.139@tcp132 [8/256/0/180] [15567.003593] LNet: Added LNI 192.168.202.139@tcp133 [8/256/0/180] [15567.997858] LNet: Added LNI 192.168.202.139@tcp134 [8/256/0/180] [15568.848804] LNet: Added LNI 192.168.202.139@tcp135 [8/256/0/180] [15576.391829] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15576.421753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15577.535845] Lustre: 464405:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.139@tcp: can't add NID 192.168.202.139@tcp130: rc = -75 [15577.694452] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15577.733900] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15577.882448] Lustre: lustre-MDT0000: new disk, initializing [15578.033737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15578.055893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15582.260769] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [15593.332701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15593.513607] Lustre: 464421:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15593.562791] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15593.569129] Lustre: Skipped 1 previous similar message [15593.627291] Lustre: lustre-MDT0001: new disk, initializing [15593.655276] Lustre: 465218:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.139@tcp: can't add NID 192.168.202.139@tcp130: rc = -75 [15593.661513] Lustre: 465218:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15593.742986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15593.773056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15593.784338] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15598.464060] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [15600.181110] Lustre: 465551:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.139@tcp: can't add NID 192.168.202.139@tcp130: rc = -75 [15600.190374] Lustre: 465551:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15608.658872] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15615.483025] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15622.983134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15623.245246] Lustre: lustre-OST0000: new disk, initializing [15623.247667] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15623.251909] Lustre: 466392:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15623.327519] Lustre: 466397:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.139@tcp: can't add NID 192.168.202.139@tcp130: rc = -75 [15623.335280] Lustre: 466397:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15623.374273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15630.123982] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [15632.965640] Lustre: 466862:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.139@tcp: can't add NID 192.168.202.139@tcp130: rc = -75 [15632.977856] Lustre: 466862:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15633.035897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15633.042173] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15633.158861] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15640.312077] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15653.860507] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15653.870638] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15653.876274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15653.887955] Lustre: Skipped 1 previous similar message [15653.918713] Lustre: Skipped 1 previous similar message [15655.470687] Lustre: server umount lustre-OST0000 complete [15660.512780] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15660.533461] LustreError: Skipped 1 previous similar message [15660.545954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15660.568610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15664.097221] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15664.104874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15664.119819] Lustre: Skipped 1 previous similar message [15665.999685] Lustre: server umount lustre-MDT0000 complete [15671.004965] LustreError: 464458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787354026 with bad export cookie 5033049190878742242 [15671.014530] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15671.024178] LustreError: 464458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15671.503575] Lustre: server umount lustre-MDT0001 complete [15690.160256] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [15694.226623] Key type lgssc unregistered [15694.576348] LNet: 468328:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15694.588309] LNetError: 468328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15694.596166] LNet: Removed LNI 192.168.202.139@tcp [15694.600467] LNet: Removed LNI 192.168.202.139@tcp100 [15695.647150] LNet: Removed LNI 192.168.202.139@tcp101 [15695.657374] LNetError: 468328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15695.662692] LNetError: 468328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15695.667372] LNet: Removed LNI 192.168.202.139@tcp102 [15695.669371] LNet: Removed LNI 192.168.202.139@tcp103 [15695.671384] LNet: Removed LNI 192.168.202.139@tcp104 [15695.673800] LNet: Removed LNI 192.168.202.139@tcp105 [15695.676033] LNet: Removed LNI 192.168.202.139@tcp106 [15695.678837] LNet: Removed LNI 192.168.202.139@tcp107 [15695.681157] LNet: Removed LNI 192.168.202.139@tcp108 [15695.685663] LNet: Removed LNI 192.168.202.139@tcp109 [15695.689179] LNet: Removed LNI 192.168.202.139@tcp110 [15695.694136] LNet: Removed LNI 192.168.202.139@tcp111 [15695.697490] LNet: Removed LNI 192.168.202.139@tcp112 [15695.703151] LNet: Removed LNI 192.168.202.139@tcp113 [15695.711193] LNet: Removed LNI 192.168.202.139@tcp114 [15695.715238] LNet: Removed LNI 192.168.202.139@tcp115 [15695.721702] LNet: Removed LNI 192.168.202.139@tcp116 [15695.723263] LNet: Removed LNI 192.168.202.139@tcp117 [15695.726614] LNet: Removed LNI 192.168.202.139@tcp118 [15695.729223] LNet: Removed LNI 192.168.202.139@tcp119 [15695.734790] LNet: Removed LNI 192.168.202.139@tcp120 [15695.745466] LNet: Removed LNI 192.168.202.139@tcp121 [15695.756076] LNet: Removed LNI 192.168.202.139@tcp122 [15695.769200] LNet: Removed LNI 192.168.202.139@tcp123 [15695.783420] LNet: Removed LNI 192.168.202.139@tcp124 [15695.791824] LNet: Removed LNI 192.168.202.139@tcp125 [15695.795258] LNet: Removed LNI 192.168.202.139@tcp126 [15695.799197] LNet: Removed LNI 192.168.202.139@tcp127 [15695.812222] LNet: Removed LNI 192.168.202.139@tcp128 [15695.816468] LNet: Removed LNI 192.168.202.139@tcp129 [15695.823702] LNet: Removed LNI 192.168.202.139@tcp130 [15695.852077] LNet: Removed LNI 192.168.202.139@tcp131 [15695.854137] LNet: Removed LNI 192.168.202.139@tcp132 [15695.872863] LNet: Removed LNI 192.168.202.139@tcp133 [15695.880182] LNet: Removed LNI 192.168.202.139@tcp134 [15695.891155] LNet: Removed LNI 192.168.202.139@tcp135 [15696.895311] Key type .llcrypt unregistered [15696.897956] Key type ._llcrypt unregistered [15724.772153] Key type ._llcrypt registered [15724.773971] Key type .llcrypt registered [15724.858832] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [15743.101430] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [15744.653433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15744.703122] alg: No test for adler32 (adler32-zlib) [15745.832231] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [15746.107386] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [15747.775204] Key type lgssc registered [15748.926795] Lustre: Echo OBD driver; http://www.lustre.org/ [15786.292969] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 19:15:40 (1787354140) [15807.847633] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [15810.753707] Key type lgssc unregistered [15811.172557] LNet: 472422:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15811.182895] LNetError: 472422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15811.200261] LNet: Removed LNI 192.168.202.139@tcp [15811.995732] Key type .llcrypt unregistered [15811.998636] Key type ._llcrypt unregistered [15833.327964] Key type ._llcrypt registered [15833.332330] Key type .llcrypt registered [15833.491526] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [15835.762075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15835.793168] alg: No test for adler32 (adler32-zlib) [15836.891424] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [15837.178832] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [15838.823513] Key type lgssc registered [15839.926816] Lustre: Echo OBD driver; http://www.lustre.org/ [15851.056585] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15851.116923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15852.489312] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15852.556699] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15852.665961] Lustre: lustre-MDT0000: new disk, initializing [15852.802812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15852.838864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15857.969716] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [15870.562071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15870.747256] Lustre: 473640:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15870.836596] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15870.850322] Lustre: Skipped 1 previous similar message [15870.991569] Lustre: lustre-MDT0001: new disk, initializing [15871.178925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15871.223684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15871.237838] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15876.873559] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [15888.271561] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15895.363808] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15903.165771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15903.671823] Lustre: lustre-OST0000: new disk, initializing [15903.675399] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15903.686795] Lustre: 475608:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15903.873206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15910.576179] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [15913.080147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15913.099335] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15913.218705] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15921.007785] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15926.531625] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing lnet_if_list [15933.506262] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15933.557100] LNet: Added LNI 192.168.202.139@tcp10 [8/256/0/180] [15942.450470] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15942.474664] LNet: Added LNI 192.168.202.139@tcp11 [8/256/0/180] [15948.633740] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15948.666579] LNet: Added LNI 192.168.202.139@tcp12 [8/256/0/180] [15955.747218] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15955.792401] LNet: Added LNI 192.168.202.139@tcp13 [8/256/0/180] [15962.045529] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15962.094302] LNet: Added LNI 192.168.202.139@tcp14 [8/256/0/180] [15967.493114] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15967.543389] LNet: Added LNI 192.168.202.139@tcp15 [8/256/0/180] [15973.400665] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15973.441886] LNet: Added LNI 192.168.202.139@tcp16 [8/256/0/180] [15979.090359] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15979.126234] LNet: Added LNI 192.168.202.139@tcp17 [8/256/0/180] [15984.770601] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15984.812369] LNet: Added LNI 192.168.202.139@tcp18 [8/256/0/180] [15990.861475] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15990.887670] LNet: Added LNI 192.168.202.139@tcp19 [8/256/0/180] [15997.202679] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15997.237267] LNet: Added LNI 192.168.202.139@tcp20 [8/256/0/180] [16003.408566] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [16003.433525] LNet: Added LNI 192.168.202.139@tcp21 [8/256/0/180] [16009.641471] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [16009.676425] LNet: Added LNI 192.168.202.139@tcp22 [8/256/0/180] [16016.318814] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [16016.356683] LNet: Added LNI 192.168.202.139@tcp23 [8/256/0/180] [16022.817474] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [16022.850146] LNet: Added LNI 192.168.202.139@tcp24 [8/256/0/180] [16029.380587] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [16029.402987] LNet: Added LNI 192.168.202.139@tcp25 [8/256/0/180] [16035.726829] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [16035.760151] LNet: Added LNI 192.168.202.139@tcp26 [8/256/0/180] [16042.986579] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [16043.019947] LNet: Added LNI 192.168.202.139@tcp27 [8/256/0/180] [16051.551674] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [16051.578896] LNet: Added LNI 192.168.202.139@tcp28 [8/256/0/180] [16058.517893] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [16058.545434] LNet: Added LNI 192.168.202.139@tcp29 [8/256/0/180] [16066.936558] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [16066.970052] LNet: Added LNI 192.168.202.139@tcp30 [8/256/0/180] [16074.138660] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [16074.204184] LNet: Added LNI 192.168.202.139@tcp31 [8/256/0/180] [16080.977945] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [16080.997833] LNet: Added LNI 192.168.202.139@tcp32 [8/256/0/180] [16088.661740] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [16088.704519] LNet: Added LNI 192.168.202.139@tcp33 [8/256/0/180] [16096.874565] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [16096.899548] LNet: Added LNI 192.168.202.139@tcp34 [8/256/0/180] [16104.770310] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [16104.789992] LNet: Added LNI 192.168.202.139@tcp35 [8/256/0/180] [16112.295901] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [16112.327658] LNet: Added LNI 192.168.202.139@tcp36 [8/256/0/180] [16119.676277] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [16119.700168] LNet: Added LNI 192.168.202.139@tcp37 [8/256/0/180] [16127.015540] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [16127.048536] LNet: Added LNI 192.168.202.139@tcp38 [8/256/0/180] [16134.674987] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [16134.705860] LNet: Added LNI 192.168.202.139@tcp39 [8/256/0/180] [16142.347804] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [16142.382882] LNet: Added LNI 192.168.202.139@tcp40 [8/256/0/180] [16152.612908] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [16152.667422] LNet: Added LNI 192.168.202.139@tcp41 [8/256/0/180] [16160.651726] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [16160.712510] LNet: Added LNI 192.168.202.139@tcp42 [8/256/0/180] [16169.296963] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [16169.362032] LNet: Added LNI 192.168.202.139@tcp43 [8/256/0/180] [16178.014416] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [16178.046901] LNet: Added LNI 192.168.202.139@tcp44 [8/256/0/180] [16185.931290] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [16185.970615] LNet: Added LNI 192.168.202.139@tcp45 [8/256/0/180] [16195.344185] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [16195.377644] LNet: Added LNI 192.168.202.139@tcp46 [8/256/0/180] [16203.912614] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [16203.945049] LNet: Added LNI 192.168.202.139@tcp47 [8/256/0/180] [16213.589119] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [16213.626968] LNet: Added LNI 192.168.202.139@tcp48 [8/256/0/180] [16222.392671] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [16222.420692] LNet: Added LNI 192.168.202.139@tcp49 [8/256/0/180] [16233.841880] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [16233.905150] LNet: Added LNI 192.168.202.139@tcp50 [8/256/0/180] [16260.577102] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16260.590938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16260.608555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16261.600295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16261.609933] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16261.625196] Lustre: Skipped 1 previous similar message [16264.109675] Lustre: server umount lustre-MDT0000 complete [16268.316029] LustreError: 473631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787354623 with bad export cookie 15690562823069580732 [16268.317573] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16268.324500] LustreError: 473631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16268.685843] Lustre: server umount lustre-MDT0001 complete [16288.223164] Lustre: 472910:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787354627/real 1787354627] req@000000005e02dec7 x1874176709412992/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787354643 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16288.277356] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16290.271161] Lustre: 472908:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787354629/real 1787354629] req@00000000ed5a6a4c x1874176709413120/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787354645 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16293.410681] Lustre: 472908:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787354632/real 1787354632] req@0000000032970985 x1874176709413248/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787354648 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16296.875641] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [16299.487193] Lustre: 472911:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787354638/real 1787354638] req@00000000836315d0 x1874176709413632/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787354654 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16299.502397] Lustre: 472911:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [16304.608501] Lustre: 472909:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787354643/real 1787354643] req@0000000016ba5751 x1874176709413760/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787354659 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16306.115832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16312.937360] LustreError: 472907:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ae749dec x1874176709414400/t0(0) o250->MGC192.168.202.139@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16313.369803] LustreError: 475602:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16313.386367] LustreError: 475602:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [16313.435399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16316.384552] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [16317.477099] LustreError: 475603:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16318.499449] LustreError: 475602:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16318.506042] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16323.558970] LustreError: 475602:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16325.913276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16326.240673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16331.319521] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [16333.310297] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16339.967784] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16347.809577] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16377.316633] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16377.323843] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16377.343961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16377.349712] Lustre: Skipped 2 previous similar messages [16379.363306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16379.394723] Lustre: Skipped 1 previous similar message [16382.353214] Lustre: server umount lustre-OST0000 complete [16387.552860] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16387.563337] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16387.578607] Lustre: Skipped 1 previous similar message [16387.586045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16393.931776] Lustre: server umount lustre-MDT0000 complete [16394.720973] LustreError: 499861:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16394.750351] LustreError: 499861:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16398.416465] LustreError: 473632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787354753 with bad export cookie 15690562823069583301 [16398.420280] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16398.431974] LustreError: 473632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16398.674873] Lustre: server umount lustre-MDT0001 complete [16410.280982] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [16413.130207] Key type lgssc unregistered [16413.546554] LNet: 502662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16413.553769] LNetError: 502662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16413.557882] LNet: Removed LNI 192.168.202.139@tcp [16413.559596] LNet: Removed LNI 192.168.202.139@tcp10 [16413.561729] LNet: Removed LNI 192.168.202.139@tcp11 [16413.564389] LNet: Removed LNI 192.168.202.139@tcp12 [16413.566091] LNet: Removed LNI 192.168.202.139@tcp13 [16413.567617] LNet: Removed LNI 192.168.202.139@tcp14 [16413.570638] LNet: Removed LNI 192.168.202.139@tcp15 [16413.572910] LNet: Removed LNI 192.168.202.139@tcp16 [16413.575639] LNet: Removed LNI 192.168.202.139@tcp17 [16413.584342] LNet: Removed LNI 192.168.202.139@tcp18 [16413.589195] LNet: Removed LNI 192.168.202.139@tcp19 [16413.594443] LNet: Removed LNI 192.168.202.139@tcp20 [16413.597971] LNet: Removed LNI 192.168.202.139@tcp21 [16413.602054] LNet: Removed LNI 192.168.202.139@tcp22 [16413.606490] LNet: Removed LNI 192.168.202.139@tcp23 [16413.613903] LNet: Removed LNI 192.168.202.139@tcp24 [16413.620459] LNet: Removed LNI 192.168.202.139@tcp25 [16413.626739] LNet: Removed LNI 192.168.202.139@tcp26 [16413.629578] LNet: Removed LNI 192.168.202.139@tcp27 [16413.634554] LNet: Removed LNI 192.168.202.139@tcp28 [16413.638523] LNet: Removed LNI 192.168.202.139@tcp29 [16413.642752] LNet: Removed LNI 192.168.202.139@tcp30 [16413.648973] LNet: Removed LNI 192.168.202.139@tcp31 [16413.659474] LNet: Removed LNI 192.168.202.139@tcp32 [16413.667309] LNet: Removed LNI 192.168.202.139@tcp33 [16413.673800] LNet: Removed LNI 192.168.202.139@tcp34 [16413.678993] LNet: Removed LNI 192.168.202.139@tcp35 [16413.687333] LNet: Removed LNI 192.168.202.139@tcp36 [16413.689818] LNet: Removed LNI 192.168.202.139@tcp37 [16413.694819] LNet: Removed LNI 192.168.202.139@tcp38 [16413.699331] LNet: Removed LNI 192.168.202.139@tcp39 [16413.704746] LNet: Removed LNI 192.168.202.139@tcp40 [16413.709890] LNet: Removed LNI 192.168.202.139@tcp41 [16413.721808] LNet: Removed LNI 192.168.202.139@tcp42 [16413.725909] LNet: Removed LNI 192.168.202.139@tcp43 [16413.735928] LNet: Removed LNI 192.168.202.139@tcp44 [16413.740938] LNet: Removed LNI 192.168.202.139@tcp45 [16413.743848] LNet: Removed LNI 192.168.202.139@tcp46 [16413.746744] LNet: Removed LNI 192.168.202.139@tcp47 [16413.758934] LNet: Removed LNI 192.168.202.139@tcp48 [16413.763358] LNet: Removed LNI 192.168.202.139@tcp49 [16413.775408] LNet: Removed LNI 192.168.202.139@tcp50 [16414.393213] Key type .llcrypt unregistered [16414.396560] Key type ._llcrypt unregistered [16436.878792] Key type ._llcrypt registered [16436.880967] Key type .llcrypt registered [16436.949703] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [16456.149345] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [16457.365233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16457.498194] alg: No test for adler32 (adler32-zlib) [16458.823720] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [16459.197853] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [16460.919219] Key type lgssc registered [16462.413759] Lustre: Echo OBD driver; http://www.lustre.org/ [16506.593684] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16506.648122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16507.942509] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16507.980716] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16508.100624] Lustre: lustre-MDT0000: new disk, initializing [16508.217916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16508.251366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16513.888686] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [16525.315892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16525.495313] Lustre: 505864:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16525.578474] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16525.584734] Lustre: Skipped 1 previous similar message [16525.724153] Lustre: lustre-MDT0001: new disk, initializing [16525.873367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16525.919277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16525.935464] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16531.221621] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [16541.411983] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16548.185992] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16556.535872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16556.810929] Lustre: *** cfs_fail_loc=90f, val=0*** [16556.888803] Lustre: lustre-OST0000: new disk, initializing [16556.893515] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16556.900507] Lustre: 507883:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16557.067662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16564.031655] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [16566.367234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16566.379969] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16566.494730] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16575.606705] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16585.186390] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16585.207610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16585.227659] Lustre: Skipped 1 previous similar message [16590.122080] Lustre: server umount lustre-OST0000 complete [16595.428045] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16595.437470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16595.450841] Lustre: Skipped 2 previous similar messages [16595.466609] Lustre: Skipped 1 previous similar message [16600.546387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16600.554512] Lustre: Skipped 1 previous similar message [16601.945905] Lustre: server umount lustre-MDT0000 complete [16606.911427] LustreError: 505856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787354962 with bad export cookie 10821139498550995455 [16606.919753] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16606.930317] LustreError: 505856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16607.323569] Lustre: server umount lustre-MDT0001 complete [16620.102971] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [16622.778142] Key type lgssc unregistered [16623.042576] LNet: 509794:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16623.049729] LNetError: 509794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16623.061162] LNet: Removed LNI 192.168.202.139@tcp [16623.941645] Key type .llcrypt unregistered [16623.944959] Key type ._llcrypt unregistered [16646.884560] Key type ._llcrypt registered [16646.887874] Key type .llcrypt registered [16647.022127] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [16648.918595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16648.954268] alg: No test for adler32 (adler32-zlib) [16649.991640] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [16650.291410] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [16651.967282] Key type lgssc registered [16653.673663] Lustre: Echo OBD driver; http://www.lustre.org/ [16662.216866] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16662.245838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16663.787548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16669.698934] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [16674.275233] LustreError: 511063:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16674.289976] LustreError: 511063:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [16678.441341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16678.803380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16683.899827] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [16692.482256] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16698.988207] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16707.482918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16708.065822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16714.657562] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [16723.502166] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16735.343759] loop0: detected capacity change from 0 to 204800000 [16735.698391] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16735.732250] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16735.764763] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16744.192458] loop0: detected capacity change from 0 to 204800000 [16744.458995] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16744.475920] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16744.500929] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16750.984605] loop0: detected capacity change from 0 to 204800000 [16758.240628] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16758.438104] Lustre: fs15246-OST0000: new disk, initializing [16758.451175] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16758.459695] Lustre: 514656:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16758.566451] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16767.084561] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [16780.545053] loop1: detected capacity change from 0 to 204800000 [16787.047628] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16787.254162] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16787.357652] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16787.451635] Lustre: fs15246-MDT0000: new disk, initializing [16787.694829] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16787.723360] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16787.884720] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16788.960163] Lustre: 515781:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000eea22207 x1874177561774080/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787355160 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16794.142718] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [16798.130826] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16798.161914] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16798.229644] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16821.221166] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16821.255621] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16824.792661] Lustre: server umount fs15246-MDT0000 complete [16826.355437] LustreError: 511058:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16831.501041] LustreError: 511060:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16834.708573] Lustre: server umount fs15246-OST0000 complete [16846.835868] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16846.843077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16846.865387] Lustre: Skipped 1 previous similar message [16851.940357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16851.948037] Lustre: Skipped 2 previous similar messages [16852.968892] Lustre: server umount lustre-OST0000 complete [16857.056273] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16857.059319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16857.086722] Lustre: Skipped 1 previous similar message [16857.105781] Lustre: Skipped 1 previous similar message [16862.176397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16862.181028] Lustre: Skipped 1 previous similar message [16863.169913] Lustre: server umount lustre-MDT0000 complete [16867.306690] LustreError: 511060:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16867.328055] LustreError: 511060:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16867.362216] LustreError: 511045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787355222 with bad export cookie 8369581005139245017 [16867.363995] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16867.367520] LustreError: 511045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16867.943433] Lustre: server umount lustre-MDT0001 complete [16882.063221] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [16885.497146] Key type lgssc unregistered [16885.880944] LNet: 518882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16885.892747] LNetError: 518882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16885.920281] LNet: Removed LNI 192.168.202.139@tcp [16887.216445] Key type .llcrypt unregistered [16887.219191] Key type ._llcrypt unregistered [16915.942966] Key type ._llcrypt registered [16915.945716] Key type .llcrypt registered [16916.057285] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [16917.839083] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16917.954633] alg: No test for adler32 (adler32-zlib) [16919.321949] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [16919.976113] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [16921.832419] Key type lgssc registered [16923.476208] Lustre: Echo OBD driver; http://www.lustre.org/ [16969.377380] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [16983.040689] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [17027.539807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17027.588547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17028.925149] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [17028.967696] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [17029.068543] Lustre: lustre-MDT0000: new disk, initializing [17029.305111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17029.351353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [17035.165269] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [17047.467778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17047.630421] Lustre: 523356:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [17047.680893] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [17047.684699] Lustre: Skipped 1 previous similar message [17047.757605] Lustre: lustre-MDT0001: new disk, initializing [17047.882035] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17047.917787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [17047.930810] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [17053.553948] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [17064.143669] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17070.220057] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17077.858407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17078.126665] Lustre: lustre-OST0000: new disk, initializing [17078.130404] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [17078.138393] Lustre: 525327:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17078.250637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17086.598233] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [17087.578648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [17087.584960] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [17087.744910] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [17099.451290] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17105.966143] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [17106.162871] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [17110.976768] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [17111.210254] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [17113.575025] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [17113.583258] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17113.592611] LustreError: Skipped 1 previous similar message [17113.596255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17113.642111] Lustre: Skipped 1 previous similar message [17118.691788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17118.703885] Lustre: Skipped 2 previous similar messages [17119.663335] Lustre: server umount lustre-OST0000 complete [17124.835115] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [17124.845994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17124.876904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17128.940791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17128.946436] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17130.731568] Lustre: server umount lustre-MDT0000 complete [17134.849252] LustreError: 525326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787355490 with bad export cookie 10148815303468071808 [17134.863389] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17134.864845] LustreError: 525326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [17135.480904] Lustre: server umount lustre-MDT0001 complete [17163.333136] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [17174.004051] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [17208.728726] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16880 sec ========= 19:39:22 (1787355562) [17210.879991] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:39:24 (1787355564) === [17215.667695] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:39:29 (1787355569) === [17266.544070] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [17270.793867] Key type lgssc unregistered [17271.524766] LNet: 531981:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17271.545223] LNetError: 531981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17271.576705] LNet: Removed LNI 192.168.202.139@tcp [17272.895473] Key type .llcrypt unregistered [17272.897055] Key type ._llcrypt unregistered