[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 485023458 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524592K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.002395] x2apic enabled [ 0.004004] Switched APIC routing to physical x2apic. [ 0.005010] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010133] LSM: Security Framework initializing [ 0.011052] Yama: becoming mindful. [ 0.013032] SELinux: Initializing. [ 0.014082] *** VALIDATE selinux *** [ 0.022653] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026795] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027185] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028131] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029105] *** VALIDATE tmpfs *** [ 0.030462] *** VALIDATE proc *** [ 0.031274] *** VALIDATE cgroup *** [ 0.032016] *** VALIDATE cgroup2 *** [ 0.033306] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034165] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036036] Spectre V2 : User space: Vulnerable [ 0.038011] Speculative Store Bypass: Vulnerable [ 0.041099] debug: unmapping init [mem 0xffffffffb9a59000-0xffffffffb9a60fff] [ 0.043274] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044826] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045033] ... version: 2 [ 0.046018] ... bit width: 48 [ 0.047018] ... generic registers: 4 [ 0.048014] ... value mask: 0000ffffffffffff [ 0.049018] ... max period: 00007fffffffffff [ 0.050016] ... fixed-purpose events: 3 [ 0.051013] ... event mask: 000000070000000f [ 0.052330] rcu: Hierarchical SRCU implementation. [ 0.054639] smp: Bringing up secondary CPUs ... [ 0.055666] x86: Booting SMP configuration: [ 0.056031] .... node #0, CPUs: #1 #2 #3 [ 0.063026] smp: Brought up 1 node, 4 CPUs [ 0.065027] smpboot: Max logical packages: 1 [ 0.066013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.121058] node 0 deferred pages initialised in 53ms [ 0.125193] devtmpfs: initialized [ 0.126280] x86/mm: Memory block size: 128MB [ 0.128868] gcov: version magic: 0x41383552 [ 0.132286] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.133088] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.134000] pinctrl core: initialized pinctrl subsystem [ 0.137176] [ 0.137581] ************************************************************* [ 0.141015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.144012] ** ** [ 0.146012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.149013] ** ** [ 0.151011] ** This means that this kernel is built to expose internal ** [ 0.153016] ** IOMMU data structures, which may compromise security on ** [ 0.155020] ** your system. ** [ 0.157010] ** ** [ 0.158010] ** If you see this message and you are not debugging the ** [ 0.161013] ** kernel, report this immediately to your vendor! ** [ 0.163014] ** ** [ 0.164010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166013] ************************************************************* [ 0.168817] NET: Registered protocol family 16 [ 0.170397] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.173059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.176076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.180122] cpuidle: using governor menu [ 0.181818] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.185507] PCI: Using configuration type 1 for base access [ 0.187208] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.195370] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.197015] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.200294] cryptd: max_cpu_qlen set to 1000 [ 0.202106] ACPI: Added _OSI(Module Device) [ 0.203108] ACPI: Added _OSI(Processor Device) [ 0.204042] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.206015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.209737] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.215509] ACPI: Interpreter enabled [ 0.216063] ACPI: PM: (supports S0 S3 S4 S5) [ 0.218013] ACPI: Using IOAPIC for interrupt routing [ 0.219124] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.222444] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.234223] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.236049] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.239026] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.243095] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.249358] acpiphp: Slot [2] registered [ 0.251126] acpiphp: Slot [5] registered [ 0.252076] acpiphp: Slot [6] registered [ 0.253086] acpiphp: Slot [7] registered [ 0.254040] acpiphp: Slot [8] registered [ 0.255020] acpiphp: Slot [9] registered [ 0.256060] acpiphp: Slot [10] registered [ 0.258142] acpiphp: Slot [3] registered [ 0.259100] acpiphp: Slot [4] registered [ 0.260074] acpiphp: Slot [11] registered [ 0.261053] acpiphp: Slot [12] registered [ 0.261940] acpiphp: Slot [13] registered [ 0.263073] acpiphp: Slot [14] registered [ 0.263873] acpiphp: Slot [15] registered [ 0.265068] acpiphp: Slot [16] registered [ 0.266085] acpiphp: Slot [17] registered [ 0.267079] acpiphp: Slot [18] registered [ 0.268098] acpiphp: Slot [19] registered [ 0.270070] acpiphp: Slot [20] registered [ 0.271113] acpiphp: Slot [21] registered [ 0.272109] acpiphp: Slot [22] registered [ 0.274110] acpiphp: Slot [23] registered [ 0.275084] acpiphp: Slot [24] registered [ 0.277086] acpiphp: Slot [25] registered [ 0.279124] acpiphp: Slot [26] registered [ 0.280495] acpiphp: Slot [27] registered [ 0.282203] acpiphp: Slot [28] registered [ 0.284207] acpiphp: Slot [29] registered [ 0.286129] acpiphp: Slot [30] registered [ 0.287058] acpiphp: Slot [31] registered [ 0.288041] PCI host bridge to bus 0000:00 [ 0.289013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.291015] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.292013] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.294066] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.295027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.298030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.299248] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.303182] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.306342] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.314011] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.317038] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.320015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.321016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.323019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.325629] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.330086] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.332049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.334837] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.340019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.352015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.356017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.363000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.374019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.379018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.393014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.403338] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.411015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.420017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.441019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.459507] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.465018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.471017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.483019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.492000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.506019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.519019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.537018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.547858] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.554022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.561019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.596023] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.610838] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.622018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.629029] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.646022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.658797] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.662436] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.664843] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.668470] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.670200] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.675033] iommu: Default domain type: Passthrough [ 0.677028] SCSI subsystem initialized [ 0.678167] ACPI: bus type USB registered [ 0.681150] usbcore: registered new interface driver usbfs [ 0.684115] usbcore: registered new interface driver hub [ 0.689191] usbcore: registered new device driver usb [ 0.691178] pps_core: LinuxPPS API ver. 1 registered [ 0.692008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.694063] PTP clock support registered [ 0.696146] EDAC MC: Ver: 3.0.0 [ 0.697195] PCI: Using ACPI for IRQ routing [ 0.698708] NetLabel: Initializing [ 0.700012] NetLabel: domain hash size = 128 [ 0.702011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.704091] NetLabel: unlabeled traffic allowed by default [ 0.707122] vgaarb: loaded [ 0.708288] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.710018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.716000] clocksource: Switched to clocksource kvm-clock [ 0.831082] VFS: Disk quotas dquot_6.6.0 [ 0.832687] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.835243] *** VALIDATE ramfs *** [ 0.836673] *** VALIDATE hugetlbfs *** [ 0.838267] pnp: PnP ACPI init [ 0.840854] pnp: PnP ACPI: found 6 devices [ 0.858236] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.862097] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.863624] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.866176] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.868862] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.871477] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.874539] NET: Registered protocol family 2 [ 0.877030] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.881701] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.885381] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.890291] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.894340] TCP: Hash tables configured (established 65536 bind 65536) [ 0.897089] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.899868] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.902554] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.904654] NET: Registered protocol family 1 [ 0.906383] RPC: Registered named UNIX socket transport module. [ 0.907795] RPC: Registered udp transport module. [ 0.908787] RPC: Registered tcp transport module. [ 0.909773] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.911228] NET: Registered protocol family 44 [ 0.912709] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.914745] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.916669] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.919279] PCI: CLS 0 bytes, default 64 [ 0.921337] Unpacking initramfs... [ 2.338919] debug: unmapping init [mem 0xffff8c86bcc54000-0xffff8c86bffbffff] [ 2.341323] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.342551] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.344158] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.840134] Initialise system trusted keyrings [ 2.841959] Key type blacklist registered [ 2.844087] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.855668] zbud: loaded [ 2.858871] *** VALIDATE nfs *** [ 2.860209] *** VALIDATE nfs4 *** [ 2.862669] pstore: using deflate compression [ 2.868704] Platform Keyring initialized [ 2.973608] NET: Registered protocol family 38 [ 2.975567] Key type asymmetric registered [ 2.977281] Asymmetric key parser 'x509' registered [ 2.978951] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.982551] io scheduler mq-deadline registered [ 2.984426] io scheduler kyber registered [ 2.986408] io scheduler bfq registered [ 2.988391] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.991297] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.994313] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.997054] ACPI: Power Button [PWRF] [ 3.002401] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.009796] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.035936] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.047929] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.072964] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.101512] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.131452] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.135906] Non-volatile memory driver v1.3 [ 3.137733] Linux agpgart interface v0.103 [ 3.169226] virtio_blk virtio1: [vda] 149784 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.172287] vda: detected capacity change from 0 to 76689408 [ 3.195764] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.197856] vdb: detected capacity change from 0 to 1073741824 [ 3.227903] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.231165] vdc: detected capacity change from 0 to 2621440000 [ 3.262612] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.265830] vdd: detected capacity change from 0 to 2621440000 [ 3.294790] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.298461] vde: detected capacity change from 0 to 4294967296 [ 3.323912] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.327588] vdf: detected capacity change from 0 to 4294967296 [ 3.343942] libphy: Fixed MDIO Bus: probed [ 3.353496] usbcore: registered new interface driver usbserial_generic [ 3.356072] usbserial: USB Serial support registered for generic [ 3.358913] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.364614] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.366917] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.370271] mousedev: PS/2 mouse device common for all mice [ 3.373505] rtc_cmos 00:05: RTC can wake from S4 [ 3.377365] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.381427] rtc_cmos 00:05: registered as rtc0 [ 3.383749] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.384203] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.387060] intel_pstate: CPU model not supported [ 3.391811] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.397582] hid: raw HID events driver (C) Jiri Kosina [ 3.399591] usbcore: registered new interface driver usbhid [ 3.401430] usbhid: USB HID core driver [ 3.402885] drop_monitor: Initializing network drop monitor service [ 3.405249] Initializing XFRM netlink socket [ 3.407345] NET: Registered protocol family 10 [ 3.410331] Segment Routing with IPv6 [ 3.411855] NET: Registered protocol family 17 [ 3.413832] mpls_gso: MPLS GSO support [ 3.419072] RAS: Correctable Errors collector initialized. [ 3.421398] AVX version of gcm_enc/dec engaged. [ 3.423157] AES CTR mode by8 optimization enabled [ 3.500110] sched_clock: Marking stable (3500015520, 0)->(4425179830, -925164310) [ 3.503459] registered taskstats version 1 [ 3.505426] Loading compiled-in X.509 certificates [ 3.507150] zswap: loaded using pool lzo/zbud [ 3.526075] Key type big_key registered [ 3.535881] Key type encrypted registered [ 3.537499] ima: No TPM chip found, activating TPM-bypass! [ 3.538912] ima: Allocated hash algorithm: sha1 [ 3.540229] ima: No architecture policies found [ 3.541264] evm: Initialising EVM extended attributes: [ 3.542611] evm: security.selinux [ 3.543477] evm: security.ima [ 3.544082] evm: security.capability [ 3.544873] evm: HMAC attrs: 0x1 [ 3.547034] rtc_cmos 00:05: setting system clock to 2026-09-06 03:12:28 UTC (1788664348) [ 3.553531] debug: unmapping init [mem 0xffffffffbaa03000-0xffffffffbabfffff] [ 3.556716] debug: unmapping init [mem 0xffffffffb9782000-0xffffffffb9a58fff] [ 3.570083] Write protecting the kernel read-only data: 28672k [ 3.573334] debug: unmapping init [mem 0xffffffffb7e03000-0xffffffffb7ffffff] [ 3.575830] debug: unmapping init [mem 0xffffffffb8714000-0xffffffffb87fffff] [ 3.610614] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.619737] systemd[1]: Detected virtualization kvm. [ 3.621465] systemd[1]: Detected architecture x86-64. [ 3.623430] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.652350] systemd[1]: No hostname configured. [ 3.654060] systemd[1]: Set hostname to . [ 3.656283] random: systemd: uninitialized urandom read (16 bytes read) [ 3.658772] systemd[1]: Initializing machine ID from random generator. [ 3.787331] random: systemd: uninitialized urandom read (16 bytes read) [ 3.790534] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.800555] random: systemd: uninitialized urandom read (16 bytes read) [ 3.803456] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.813160] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.328066] device-mapper: uevent: version 1.0.3 [ 4.330606] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ 5.018814] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon.[ 5.044811] random: fast init done [ 5.102159] scsi host0: ata_piix [ 5.118177] scsi host1: ata_piix [ 5.119561] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.121320] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.770277] dracut-initqueue[580]: RTNETLINK answers: File exists [ 10.066434] random: crng init done [ 10.067825] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.397923] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.614628] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.889784] SELinux: Disabled at runtime. [ 11.960471] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.968800] systemd[1]: Detected virtualization kvm. [ 11.971230] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.416587] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.419278] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.423252] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.427389] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.430298] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.439274] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.449760] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] [ 12.597576] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.999095] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.415097] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.441549] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.554086] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.570583] EDAC sbridge: Ver: 1.1.2 [ 15.116094] Key type dns_resolver registered [ 15.455730] NFS: Registering the id_resolver key type [ 15.457315] Key type id_resolver registered [ 15.458546] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg225-server login: [ 42.413639] libcfs: loading out-of-tree module taints kernel. [ 42.434660] Key type ._llcrypt registered [ 42.437077] Key type .llcrypt registered [ 42.482816] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_hostid [ 50.750794] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 51.377975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 51.388411] alg: No test for adler32 (adler32-zlib) [ 52.414061] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 52.766210] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 54.399201] Key type lgssc registered [ 56.572885] Lustre: Echo OBD driver; http://www.lustre.org/ [ 75.484345] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 103.455245] hrtimer: interrupt took 2352491 ns [ 122.571769] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 137.000904] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 137.058431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 138.348733] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 138.406510] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 138.523975] Lustre: lustre-MDT0000: new disk, initializing [ 138.648943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 138.680087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 143.945262] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 158.313821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 158.419553] Lustre: 6486:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 158.447373] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 158.451637] Lustre: Skipped 1 previous similar message [ 158.536477] Lustre: lustre-MDT0001: new disk, initializing [ 158.605471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 158.634165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 158.656161] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 163.265367] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 168.233085] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 178.887670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 179.298449] Lustre: lustre-OST0000: new disk, initializing [ 179.306418] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 179.320962] Lustre: 8423:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 179.459590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 181.621082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 181.639696] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 181.711930] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 186.878909] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 203.316816] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 203.458730] Lustre: lustre-OST0001: new disk, initializing [ 203.462660] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 203.467855] Lustre: 9495:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 203.553996] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 210.531390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 210.561952] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 210.605210] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 212.125734] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 224.678762] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 230.927796] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 238.586078] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing check_logdir /tmp/testlogs/ [ 244.447924] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing yml_node [ 248.624459] Lustre: DEBUG MARKER: Client: 2.17.58.4 [ 251.509909] Lustre: DEBUG MARKER: MDS: 2.17.58.4 [ 254.347500] Lustre: DEBUG MARKER: OSS: 2.17.58.4 [ 255.929424] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 5 23:16:40 EDT 2026 [ 273.991854] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 275.974180] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 292.837902] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 292.843616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 292.863388] Lustre: Skipped 1 previous similar message [ 292.897105] Lustre: Skipped 3 previous similar messages [ 297.957764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 299.414837] Lustre: server umount lustre-MDT0000 complete [ 303.878467] LustreError: 9493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788664648 with bad export cookie 6674583293729286566 [ 303.887129] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 303.894140] LustreError: 9493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 308.193888] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 308.195807] LustreError: 10075:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 308.227860] Lustre: Skipped 2 previous similar messages [ 308.228410] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 308.228416] Lustre: Skipped 3 previous similar messages [ 308.279099] LustreError: 10075:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 310.447762] Lustre: server umount lustre-MDT0001 complete [ 315.923356] Lustre: server umount lustre-OST0000 complete [ 320.514465] Lustre: server umount lustre-OST0001 complete [ 329.322114] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_hostid [ 339.788434] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 379.254110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 379.825162] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 379.886825] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 380.029407] Lustre: lustre-MDT0000: new disk, initializing [ 380.209008] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 380.243146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 385.460179] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 396.850390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 397.003067] Lustre: 15058:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 397.072856] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 397.077925] Lustre: Skipped 1 previous similar message [ 397.204369] Lustre: lustre-MDT0001: new disk, initializing [ 397.341564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 397.381485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 397.392160] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 402.529338] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 412.944681] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 419.430846] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 427.212640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 427.541778] Lustre: lustre-OST0000: new disk, initializing [ 427.547079] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 427.557844] Lustre: 17024:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 427.687396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 428.900981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 428.914893] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 429.111716] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 434.458554] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 444.474646] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 448.923366] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 449.138390] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 454.757972] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 455.016902] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 459.744543] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 459.755614] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 459.775564] LustreError: Skipped 1 previous similar message [ 459.786892] Lustre: Skipped 1 previous similar message [ 459.789267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 459.807855] Lustre: Skipped 2 previous similar messages [ 463.834413] Lustre: server umount lustre-OST0000 complete [ 468.960783] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 468.970435] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 468.987026] Lustre: Skipped 1 previous similar message [ 469.010865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 474.612223] Lustre: server umount lustre-MDT0000 complete [ 475.105639] LustreError: 15064:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 475.120622] LustreError: 15064:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 478.164108] LustreError: 15049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788664823 with bad export cookie 6674583293729288099 [ 478.175734] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 478.186972] LustreError: 15049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 478.686956] Lustre: server umount lustre-MDT0001 complete [ 483.414323] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 23:20:27 (1788664827) [ 486.583898] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 488.433395] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 23:20:32 (1788664832) [ 495.368920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 495.997128] LustreError: 19435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 496.162503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 501.231163] LustreError: 19435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 501.250478] LustreError: 19435:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 501.512074] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 505.322091] LustreError: 19436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 510.929211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 516.003330] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 523.132954] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 528.117022] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 534.628523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 541.786591] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 548.208690] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 556.002436] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 556.010969] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 556.012332] Lustre: Skipped 1 previous similar message [ 556.020486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 556.020495] Lustre: Skipped 2 previous similar messages [ 560.360298] Lustre: server umount lustre-OST0000 complete [ 566.243771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 566.260963] Lustre: Skipped 1 previous similar message [ 570.306948] Lustre: server umount lustre-MDT0000 complete [ 571.360332] LustreError: 19435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 571.369251] LustreError: 19435:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 573.961114] LustreError: 19415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788664918 with bad export cookie 6674583293729288981 [ 573.967209] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 573.979472] LustreError: 19415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 574.371879] Lustre: server umount lustre-MDT0001 complete [ 586.356911] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 588.958898] Key type lgssc unregistered [ 589.395494] LNet: 22967:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 589.411437] LNetError: 22967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 589.433972] LNet: Removed LNI 192.168.202.125@tcp [ 590.779160] Key type .llcrypt unregistered [ 590.781497] Key type ._llcrypt unregistered [ 613.344852] Key type ._llcrypt registered [ 613.349153] Key type .llcrypt registered [ 613.417971] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 614.717078] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 614.751563] alg: No test for adler32 (adler32-zlib) [ 615.819290] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 616.075407] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 617.759770] Key type lgssc registered [ 618.957210] Lustre: Echo OBD driver; http://www.lustre.org/ [ 626.722333] loop: module loaded [ 626.833417] loop0: detected capacity change from 0 to 204800000 [ 626.995825] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 627.007644] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 627.242713] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 632.050125] loop0: detected capacity change from 0 to 204800000 [ 632.187843] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 632.194508] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 632.204521] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 636.723150] loop0: detected capacity change from 0 to 204800000 [ 641.997711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 642.026433] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 643.287813] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 643.349630] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 643.407924] Lustre: lustre-MDT0000: new disk, initializing [ 643.536751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 643.561707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 647.235586] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 654.716458] loop1: detected capacity change from 0 to 204800000 [ 659.372929] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 659.644259] Lustre: lustre-OST07c6: new disk, initializing [ 659.648638] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 659.653369] Lustre: Skipped 1 previous similar message [ 659.659787] Lustre: 26148:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 659.752219] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 665.395826] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 667.279730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 667.301303] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 667.354536] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 670.756729] Lustre: 25135:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 670.761075] Lustre: 25135:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 670.766515] Lustre: 25135:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 670.775904] Lustre: 25135:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 670.788928] Lustre: 25135:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 670.794296] Lustre: 25135:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 687.939498] Lustre: 25136:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 687.966764] Lustre: 25136:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 687.978383] Lustre: 25136:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 687.982832] Lustre: 25136:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 687.988626] Lustre: 25136:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 687.994513] Lustre: 25136:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 692.510294] Lustre: 25136:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 692.515488] Lustre: 25136:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 692.520723] Lustre: 25136:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 692.526115] Lustre: 25136:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 692.531241] Lustre: 25136:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 692.536315] Lustre: 25136:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 692.544738] Lustre: 25136:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 692.549968] Lustre: 25136:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 692.555141] Lustre: 25136:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 692.560626] Lustre: 25136:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 692.565639] Lustre: 25136:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 692.572113] Lustre: 25136:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 695.720892] Lustre: 25135:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 695.748840] Lustre: 25135:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 695.752182] Lustre: 25135:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 695.758412] Lustre: 25135:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 695.763484] Lustre: 25135:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 695.768534] Lustre: 25135:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 695.773484] Lustre: 25135:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 695.778753] Lustre: 25135:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 695.784714] Lustre: 25135:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 695.789831] Lustre: 25135:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 695.795496] Lustre: 25135:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 695.800261] Lustre: 25135:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 700.402923] Lustre: 25137:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 700.412268] Lustre: 25137:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 700.419275] Lustre: 25137:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 700.424421] Lustre: 25137:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 700.431473] Lustre: 25137:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 700.438098] Lustre: 25137:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 700.445369] Lustre: 25137:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 700.454450] Lustre: 25137:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 700.458227] Lustre: 25137:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 700.461240] Lustre: 25137:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 700.464618] Lustre: 25137:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 700.469330] Lustre: 25137:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 711.605808] Lustre: 25135:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 711.616881] Lustre: 25135:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 711.622858] Lustre: 25135:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 711.633476] Lustre: 25135:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 711.646177] Lustre: 25135:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 711.662034] Lustre: 25135:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 711.676965] Lustre: 25135:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 711.688356] Lustre: 25135:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 711.701654] Lustre: 25135:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 711.718844] Lustre: 25135:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 711.741808] Lustre: 25135:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 711.755843] Lustre: 25135:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 764.385257] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 764.395156] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 769.509831] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 774.632105] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 775.135534] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 775.255980] Lustre: server umount lustre-OST07c6 complete [ 785.087977] Lustre: server umount lustre-MDT0000 complete [ 803.575903] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 806.061857] Key type lgssc unregistered [ 806.249886] LNet: 28707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 806.254791] LNetError: 28707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 806.262650] LNet: Removed LNI 192.168.202.125@tcp [ 806.988151] Key type .llcrypt unregistered [ 806.999148] Key type ._llcrypt unregistered [ 825.487377] Key type ._llcrypt registered [ 825.491709] Key type .llcrypt registered [ 825.630534] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 826.785478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 826.878641] alg: No test for adler32 (adler32-zlib) [ 828.036611] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 828.276207] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 829.967643] Key type lgssc registered [ 831.190344] Lustre: Echo OBD driver; http://www.lustre.org/ [ 838.725572] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 838.739191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 840.228084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 844.733255] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 850.403488] LustreError: 29974:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 850.418621] LustreError: 29974:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 853.063365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 853.369684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 857.785119] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 865.201589] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 870.653749] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 876.963077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 877.185208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 883.014870] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 885.370494] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 889.884746] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 904.159845] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 904.164668] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 904.175178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 906.220167] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 906.223734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 906.228907] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 906.253028] Lustre: Skipped 1 previous similar message [ 908.827526] Lustre: server umount lustre-OST0000 complete [ 914.912685] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 914.919244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 914.934861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 919.356140] Lustre: server umount lustre-MDT0000 complete [ 921.573514] LustreError: 29975:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 921.600480] LustreError: 29975:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 922.878804] LustreError: 29956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788665267 with bad export cookie 6907866752854158577 [ 922.890355] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 922.909343] LustreError: 29956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 923.134971] Lustre: server umount lustre-MDT0001 complete [ 935.005598] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 937.566588] Key type lgssc unregistered [ 937.946071] LNet: 33707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 937.952039] LNetError: 33707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 937.968771] LNet: Removed LNI 192.168.202.125@tcp [ 938.841389] Key type .llcrypt unregistered [ 938.844616] Key type ._llcrypt unregistered [ 958.081101] Key type ._llcrypt registered [ 958.082612] Key type .llcrypt registered [ 958.216203] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 959.712638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 959.790934] alg: No test for adler32 (adler32-zlib) [ 960.838973] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 961.059712] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 962.751436] Key type lgssc registered [ 963.985712] Lustre: Echo OBD driver; http://www.lustre.org/ [ 971.028989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 971.042820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 972.472382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 977.409336] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 983.011837] LustreError: 34977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 983.046466] LustreError: 34977:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 985.711187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 986.052102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 990.175166] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 997.796398] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1003.897777] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1011.615210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1012.139996] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1017.195058] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1019.223570] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 1026.604821] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1040.866909] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1040.890660] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1040.901014] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1040.906711] Lustre: Skipped 1 previous similar message [ 1042.145132] Lustre: server umount lustre-OST0000 complete [ 1045.986296] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1046.000906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1046.004346] Lustre: Skipped 1 previous similar message [ 1046.014486] Lustre: Skipped 2 previous similar messages [ 1051.106894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1051.117324] Lustre: Skipped 1 previous similar message [ 1051.900771] Lustre: server umount lustre-MDT0000 complete [ 1055.543033] LustreError: 34957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788665400 with bad export cookie 11830398521625744180 [ 1055.550478] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1055.554255] LustreError: 34957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1055.998179] Lustre: server umount lustre-MDT0001 complete [ 1066.552485] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 1070.114288] Key type lgssc unregistered [ 1070.432681] LNet: 38533:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1070.438418] LNetError: 38533:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1070.462515] LNet: Removed LNI 192.168.202.125@tcp [ 1071.457948] Key type .llcrypt unregistered [ 1071.459583] Key type ._llcrypt unregistered [ 1095.015547] Key type ._llcrypt registered [ 1095.018256] Key type .llcrypt registered [ 1095.107180] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 1096.121531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1096.202724] alg: No test for adler32 (adler32-zlib) [ 1097.367818] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 1097.622960] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 1099.279532] Key type lgssc registered [ 1100.488519] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1108.075592] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1108.088496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1109.510687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1113.932451] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 1119.716538] LustreError: 39948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1119.739598] LustreError: 39948:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1121.811942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1122.221056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1126.727463] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 1133.692400] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1140.330816] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1148.450201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1148.837734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1154.986258] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1155.445612] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 1163.513873] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1176.357516] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.25@tcp (stopping) [ 1177.568689] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1177.578232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1177.587841] Lustre: Skipped 2 previous similar messages [ 1177.596362] Lustre: Skipped 2 previous similar messages [ 1179.978293] Lustre: server umount lustre-MDT0000 complete [ 1181.430865] LustreError: 39944:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1182.696859] LustreError: 39944:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1182.714515] LustreError: 39944:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1183.410111] LustreError: 39928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788665528 with bad export cookie 11414518201205928360 [ 1183.425785] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1186.554975] LustreError: 39943:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1186.600670] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.25@tcp (stopping) [ 1187.809046] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1188.833743] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1188.845361] Lustre: Skipped 1 previous similar message [ 1189.872942] Lustre: server umount lustre-MDT0001 complete [ 1204.940695] Lustre: server umount lustre-OST0000 complete [ 1222.088393] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 1224.922703] Key type lgssc unregistered [ 1225.336785] LNet: 43806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1225.340462] LNetError: 43806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1225.363831] LNet: Removed LNI 192.168.202.125@tcp [ 1226.260162] Key type .llcrypt unregistered [ 1226.261867] Key type ._llcrypt unregistered [ 1252.676606] Key type ._llcrypt registered [ 1252.680833] Key type .llcrypt registered [ 1252.850334] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 1254.282753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1254.353207] alg: No test for adler32 (adler32-zlib) [ 1255.303753] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 1255.473610] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 1257.095224] Key type lgssc registered [ 1258.930183] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1265.724705] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1265.745248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1267.098180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1271.218967] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 1277.414116] LustreError: 45220:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1277.429621] LustreError: 45220:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1278.896599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1279.147086] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1283.753681] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 1290.521559] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1294.843055] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1300.636260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1300.915752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1306.094872] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 1312.241911] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1314.623721] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1321.266369] Lustre: lustre-OST0000: Not available for connect from 192.168.202.25@tcp (stopping) [ 1322.466777] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1322.472041] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1322.489337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1322.531332] Lustre: Skipped 1 previous similar message [ 1326.327818] Lustre: lustre-OST0000: Not available for connect from 192.168.202.25@tcp (stopping) [ 1326.338837] Lustre: Skipped 1 previous similar message [ 1327.379820] Lustre: server umount lustre-OST0000 complete [ 1346.016868] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1346.025993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1346.042567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1348.064124] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1353.184594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1353.192257] Lustre: Skipped 3 previous similar messages [ 1360.351173] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1360.565740] Lustre: server umount lustre-MDT0000 complete [ 1362.407492] LustreError: 45949:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1362.422725] LustreError: 45949:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1364.868967] LustreError: 45202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788665709 with bad export cookie 4253038221660395246 [ 1364.870721] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1364.877376] LustreError: 45202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1379.295475] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1379.625829] Lustre: server umount lustre-MDT0001 complete [ 1390.705267] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 1393.683765] Key type lgssc unregistered [ 1393.905758] LNet: 48932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1393.925410] LNetError: 48932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1393.945883] LNet: Removed LNI 192.168.202.125@tcp [ 1394.781536] Key type .llcrypt unregistered [ 1394.787908] Key type ._llcrypt unregistered [ 1417.316517] Key type ._llcrypt registered [ 1417.321401] Key type .llcrypt registered [ 1417.452741] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 1418.873530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1418.903467] alg: No test for adler32 (adler32-zlib) [ 1419.984845] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 1420.244786] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 1421.919150] Key type lgssc registered [ 1423.166460] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1430.477175] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1430.490629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1431.942485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1436.178366] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 1441.249634] LustreError: 50346:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1441.264723] LustreError: 50346:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1443.896120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1444.133971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1449.111265] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 1457.711418] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1463.357240] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1469.460778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1469.697288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1475.499713] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 1476.866859] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1482.498791] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1487.616452] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1490.972490] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1503.076356] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-0523h37m27s [ 1504.247857] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1507.815618] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1507.821168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1507.830542] Lustre: Skipped 1 previous similar message [ 1507.845122] Lustre: Skipped 2 previous similar messages [ 1508.922879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1508.929210] Lustre: Skipped 2 previous similar messages [ 1510.723152] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.25@tcp (stopping) [ 1510.733935] Lustre: Skipped 1 previous similar message [ 1513.081735] Lustre: server umount lustre-MDT0000 complete [ 1514.984101] LustreError: 50346:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1515.022189] LustreError: 50346:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1516.696720] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1521.124285] LustreError: 50343:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1521.156558] LustreError: 50343:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1523.045349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1523.142647] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1523.383381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1523.463619] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1524.403767] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1527.688974] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 1530.899233] Lustre: DEBUG MARKER: Wait for df (36676) ... [ 1532.477162] Lustre: DEBUG MARKER: done [ 1537.505878] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1537.513981] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1537.535356] Lustre: Skipped 1 previous similar message [ 1537.544471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1539.043090] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1539.056100] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1545.187635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1545.200923] Lustre: Skipped 4 previous similar messages [ 1550.303119] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1550.529895] Lustre: server umount lustre-OST0000 complete [ 1555.425739] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1555.429920] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1555.431130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1555.431136] Lustre: Skipped 2 previous similar messages [ 1555.448735] Lustre: Skipped 1 previous similar message [ 1568.223231] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1568.599439] Lustre: server umount lustre-MDT0000 complete [ 1572.726558] LustreError: 52189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788665917 with bad export cookie 7471043377503663132 [ 1572.732577] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1572.744558] LustreError: 52189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1573.097745] Lustre: server umount lustre-MDT0001 complete [ 1585.434269] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 1588.085817] Key type lgssc unregistered [ 1588.388828] LNet: 55272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1588.396702] LNetError: 55272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1588.409800] LNet: Removed LNI 192.168.202.125@tcp [ 1589.642638] Key type .llcrypt unregistered [ 1589.643981] Key type ._llcrypt unregistered [ 1630.717211] Key type ._llcrypt registered [ 1630.720793] Key type .llcrypt registered [ 1630.882321] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 1632.217921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1632.245928] alg: No test for adler32 (adler32-zlib) [ 1633.448555] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 1633.792442] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 1635.479230] Key type lgssc registered [ 1636.692958] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1644.587048] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1644.620335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1645.942154] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1645.982940] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1646.102040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1651.063904] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 1662.591699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1662.775176] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1662.851631] Lustre: 57478:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1663.357824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1669.187748] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 1677.254317] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1683.153481] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1689.065316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1689.283850] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1689.558566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1696.064465] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 1698.989232] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1703.695237] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1711.177198] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-0523h40m55s [ 1713.026772] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1720.573768] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1722.933945] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1722.993302] LustreError: 59629:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1743.135406] Lustre: lustre-MDT0000: Client 30549c8e-0bcf-4b3a-b039-6b58a096db63 (at 192.168.202.25@tcp) reconnecting [ 1763.095434] LustreError: 59629:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1765.459511] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1768.113618] Lustre: DEBUG MARKER: done [ 1775.599078] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1775.604430] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1775.686232] Lustre: server umount lustre-OST0000 complete [ 1781.233072] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1781.233695] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1781.235113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1781.264521] Lustre: Skipped 1 previous similar message [ 1782.305690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1782.312912] Lustre: Skipped 2 previous similar messages [ 1788.388557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1788.398830] Lustre: Skipped 1 previous similar message [ 1789.508329] Lustre: server umount lustre-MDT0000 complete [ 1792.490623] LustreError: 59629:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1792.510524] LustreError: 59629:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1793.581928] LustreError: 57470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788666138 with bad export cookie 4470423211612446671 [ 1793.598089] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1793.603729] LustreError: 57470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1793.958083] Lustre: server umount lustre-MDT0001 complete [ 1804.069548] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 1806.358694] Key type lgssc unregistered [ 1806.588468] LNet: 61803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1806.596128] LNetError: 61803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1806.626905] LNet: Removed LNI 192.168.202.125@tcp [ 1807.464633] Key type .llcrypt unregistered [ 1807.465906] Key type ._llcrypt unregistered [ 1849.875830] Key type ._llcrypt registered [ 1849.884753] Key type .llcrypt registered [ 1850.081134] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 1851.759677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1851.850543] alg: No test for adler32 (adler32-zlib) [ 1853.015989] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 1853.399822] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 1855.207228] Key type lgssc registered [ 1857.119494] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1861.618428] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1861.648906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1862.972544] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1863.026957] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1863.162931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1864.294885] Lustre: Failing over lustre-MDT0000 [ 1864.827493] Lustre: server umount lustre-MDT0000 complete [ 1871.133368] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:43:35 (1788666215) [ 1877.263560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1877.687731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1881.673494] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 1890.662695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1890.748669] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1890.786926] Lustre: 64600:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1891.062419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1896.220508] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 1904.004983] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1909.788643] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1916.074521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1916.284113] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1916.463680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1921.780324] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 1922.198550] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1928.554988] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1932.702032] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1938.731452] Lustre: server umount lustre-MDT0000 complete [ 1942.300283] LustreError: 64591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788666287 with bad export cookie 13848333625482038820 [ 1942.307588] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1942.315572] LustreError: 64591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1942.589310] Lustre: server umount lustre-MDT0001 complete [ 1946.045778] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1950.495858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1957.706460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1958.351793] Lustre: 63521:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788666287/real 1788666287] req@000000004ca9dccc x1875552447270656/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788666303 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1958.387484] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1958.879828] Lustre: 63518:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788666287/real 1788666287] req@00000000f1b4cc90 x1875552447270784/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788666303 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1958.907312] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1962.527360] Lustre: 63520:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788666291/real 1788666291] req@00000000c91f7de0 x1875552447271040/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788666307 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1967.138912] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xc02f2a0f8df4570e [ 1967.158881] Lustre: MGC192.168.202.125@tcp: Connection restored to 0@lo (at 0@lo) [ 1967.741086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1967.888664] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1968.618179] Lustre: 63521:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788666297/real 1788666297] req@00000000d21353cd x1875552447271168/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788666313 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1968.648078] Lustre: 63521:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1968.750108] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1972.406257] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 1973.857192] Lustre: 63518:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788666302/real 1788666302] req@00000000d39cf0e4 x1875552447271552/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788666318 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1973.882203] Lustre: 63518:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1974.253657] LustreError: 66448:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1974.287749] LustreError: 66448:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 1979.363620] LustreError: 68366:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1979.385226] LustreError: 68366:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1980.926866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1981.439769] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1985.752341] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 1986.535833] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1993.121220] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1998.291938] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2006.143386] Lustre: server umount lustre-MDT0000 complete [ 2007.010915] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2007.017723] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2007.023454] LustreError: 66448:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2007.034101] Lustre: Skipped 1 previous similar message [ 2009.477443] LustreError: 64591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788666354 with bad export cookie 13848333625482041102 [ 2009.486038] LustreError: 64591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2009.487581] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2009.741026] Lustre: server umount lustre-MDT0001 complete [ 2015.632279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2025.326473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2028.521440] Lustre: 63518:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788666357/real 1788666357] req@0000000080df0c51 x1875552447297920/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788666373 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2028.567968] Lustre: 63518:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2028.579335] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2028.605127] Lustre: Skipped 1 previous similar message [ 2034.784506] LustreError: 63517:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000043896213 x1875552447299072/t0(0) o250->MGC192.168.202.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2035.178762] LustreError: 66447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2035.198859] LustreError: 66447:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2035.264834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2035.318095] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2039.380307] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 2043.360769] LustreError: 71048:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2043.375362] LustreError: 71048:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2047.034700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2048.363407] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2051.410382] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 2057.823179] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2062.527172] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2068.965069] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2068.968413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2068.987709] Lustre: Skipped 2 previous similar messages [ 2069.001870] Lustre: Skipped 2 previous similar messages [ 2069.253137] Lustre: server umount lustre-MDT0000 complete [ 2072.566476] LustreError: 64593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788666417 with bad export cookie 13848333625482042481 [ 2072.574743] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2072.577356] LustreError: 64593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2072.820205] Lustre: server umount lustre-MDT0001 complete [ 2078.976963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2082.452549] Lustre: DEBUG MARKER: files compared the same [ 2094.991712] Lustre: server umount lustre-OST0000 complete [ 2109.751776] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 2112.515276] Key type lgssc unregistered [ 2112.848406] LNet: 74321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2112.860222] LNetError: 74321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2112.888653] LNet: Removed LNI 192.168.202.125@tcp [ 2113.843437] Key type .llcrypt unregistered [ 2113.849191] Key type ._llcrypt unregistered [ 2135.890927] Key type ._llcrypt registered [ 2135.895858] Key type .llcrypt registered [ 2136.060326] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 2137.213546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2137.233498] alg: No test for adler32 (adler32-zlib) [ 2138.282071] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 2138.590620] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 2140.327288] Key type lgssc registered [ 2141.893772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2149.272511] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2149.303127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2150.835562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2155.380798] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug +malloc all [ 2160.033496] LustreError: 75736:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2160.050862] LustreError: 75736:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2161.062134] LustreError: 75737:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2163.140527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2163.491307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2168.075941] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug +malloc all [ 2175.443515] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2180.380793] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2186.869303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2187.270152] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2192.824542] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug +malloc all [ 2196.524680] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2200.568606] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2211.818787] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2211.826979] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2211.856345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2215.901800] Lustre: server umount lustre-OST0000 complete [ 2222.049783] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2222.055843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2222.057747] Lustre: Skipped 2 previous similar messages [ 2222.062891] Lustre: Skipped 1 previous similar message [ 2226.667473] Lustre: server umount lustre-MDT0000 complete [ 2227.168800] LustreError: 75737:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2227.180485] LustreError: 75737:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2230.458044] LustreError: 75716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788666575 with bad export cookie 10322863791585203668 [ 2230.463908] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2230.464976] LustreError: 75716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2230.769745] Lustre: server umount lustre-MDT0001 complete [ 2244.082688] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 2246.782783] Key type lgssc unregistered [ 2247.055718] LNet: 79266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2247.065713] LNetError: 79266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2248.111347] LNet: Removed LNI 192.168.202.125@tcp [ 2249.184674] Key type .llcrypt unregistered [ 2249.192322] Key type ._llcrypt unregistered [ 2278.021912] Key type ._llcrypt registered [ 2278.024975] Key type .llcrypt registered [ 2278.129459] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 2278.849639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2278.886083] alg: No test for adler32 (adler32-zlib) [ 2279.992098] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 2280.212707] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 2281.887268] Key type lgssc registered [ 2283.470761] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2293.428947] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2293.442680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2309.151585] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2314.272553] LustreError: 80655:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.125@tcp: failed processing log, type 4: rc = -110 [ 2340.959648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2346.864062] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 2354.330723] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2362.570721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2362.657228] Lustre: *** cfs_fail_loc=706, val=0*** [ 2362.934652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2363.004798] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2367.132489] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 2367.975810] LustreError: 80681:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2368.000265] LustreError: 80681:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2373.092450] LustreError: 80680:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2374.974457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2375.147302] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2375.155049] Lustre: Skipped 1 previous similar message [ 2375.331992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2379.656378] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 2386.617482] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2392.091151] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2396.136084] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2396.154075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2396.158607] Lustre: Skipped 1 previous similar message [ 2396.176479] Lustre: Skipped 1 previous similar message [ 2400.883460] Lustre: server umount lustre-OST0000 complete [ 2405.858670] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2405.868970] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2405.885145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2411.488473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2411.503272] Lustre: Skipped 3 previous similar messages [ 2412.034461] Lustre: server umount lustre-MDT0000 complete [ 2416.406819] LustreError: 80663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788666761 with bad export cookie 6866702160451900306 [ 2416.423296] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2416.430070] LustreError: 80663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2416.739546] Lustre: server umount lustre-MDT0001 complete [ 2429.132958] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 2433.083968] Key type lgssc unregistered [ 2433.386767] LNet: 84269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2433.392068] LNetError: 84269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2433.410782] LNet: Removed LNI 192.168.202.125@tcp [ 2434.393210] Key type .llcrypt unregistered [ 2434.397835] Key type ._llcrypt unregistered [ 2463.469069] Key type ._llcrypt registered [ 2463.471765] Key type .llcrypt registered [ 2463.608218] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 2464.828314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2464.890467] alg: No test for adler32 (adler32-zlib) [ 2466.236645] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 2466.612996] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 2468.295188] Key type lgssc registered [ 2469.247776] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2477.950666] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2477.968725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2484.500909] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 2493.158308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2493.581065] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2499.041143] LustreError: 86371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2499.054333] LustreError: 86371:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2499.304642] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 2503.143448] LustreError: 86370:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2503.159155] LustreError: 86370:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2508.261256] LustreError: 86374:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2508.299314] LustreError: 86374:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2509.842579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2510.112606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2515.426705] LustreError: 86369:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2515.465823] LustreError: 86369:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2517.719808] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 2523.641151] LustreError: 86375:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2523.668038] LustreError: 86375:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2527.685566] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2527.703400] Lustre: MGS: Client c0f12c6a-8287-4e4d-b600-1e1608f8433f (at 0@lo) reconnecting [ 2527.734510] LustreError: 86385:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2528.110330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2528.227364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2533.916495] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 2552.190758] Lustre: server umount lustre-OST0000 complete [ 2558.944356] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2558.945943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2558.958731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2559.006613] Lustre: Skipped 1 previous similar message [ 2564.075347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2564.081838] Lustre: Skipped 2 previous similar messages [ 2569.183257] Lustre: 85045:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788666898/real 1788666898] req@000000009e1adb9d x1875553090039552/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788666914 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2569.185724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2569.208397] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2569.230643] Lustre: Skipped 1 previous similar message [ 2570.207490] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2570.271168] Lustre: 85045:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788666899/real 1788666899] req@00000000e613aaf6 x1875553090040192/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788666915 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2570.306714] Lustre: server umount lustre-MDT0000 complete [ 2574.499195] Lustre: server umount lustre-MDT0001 complete [ 2578.239223] Lustre: server umount MGS complete [ 2589.611822] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 23:55:34 (1788666934) [ 2622.073812] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_hostid [ 2633.354311] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 2669.081783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2673.887889] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 2683.681388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2683.938298] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2683.967536] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2684.052806] Lustre: lustre-MDT0001: new disk, initializing [ 2684.172437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2684.217757] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2684.229124] Lustre: 93563:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2684.244063] LustreError: 93563:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2689.128063] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 2699.175051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2699.415684] Lustre: lustre-OST0000: new disk, initializing [ 2699.420269] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2699.427699] Lustre: 94329:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2699.513635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2704.610699] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 2707.552541] Lustre: 94785:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000006d3e294e x1875553090054144/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788667068 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2713.838376] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2721.211953] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2721.224992] Lustre: MGS: Client 508cf4cb-db1d-43d6-97f0-068902b5a735 (at 0@lo) reconnecting [ 2721.315441] Lustre: 92831:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2721.338918] LustreError: 93549:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2721.378430] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2721.454377] Lustre: lustre-MDT0000: new disk, initializing [ 2721.614983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2721.639769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2721.760507] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2722.730278] Lustre: 95398:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000c1897863 x1875553090060672/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788667083 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2727.728226] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 2727.788439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2727.796522] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2727.989593] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2753.504118] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2753.512511] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2753.521692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2753.527773] Lustre: Skipped 1 previous similar message [ 2753.539937] Lustre: Skipped 1 previous similar message [ 2758.440362] Lustre: server umount lustre-OST0000 complete [ 2763.744923] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2763.774727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2763.782577] Lustre: Skipped 1 previous similar message [ 2768.441928] Lustre: server umount lustre-MDT0000 complete [ 2768.870277] LustreError: 93534:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2768.887853] LustreError: 93534:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2787.295131] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2787.680074] Lustre: server umount lustre-MDT0001 complete [ 2792.028496] Lustre: server umount MGS complete [ 2804.321527] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 23:59:08 (1788667148) [ 2821.081511] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 2823.746462] Key type lgssc unregistered [ 2824.013122] LNet: 97896:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2824.018885] LNetError: 97896:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2825.068837] LNet: Removed LNI 192.168.202.125@tcp [ 2826.003521] Key type .llcrypt unregistered [ 2826.018350] Key type ._llcrypt unregistered [ 2845.546422] Key type ._llcrypt registered [ 2845.548947] Key type .llcrypt registered [ 2845.706089] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 2846.599628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2846.640960] alg: No test for adler32 (adler32-zlib) [ 2847.841769] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 2848.056392] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 2849.783292] Key type lgssc registered [ 2851.327111] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2859.941246] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2859.957936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2861.634967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2866.735848] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 2871.778350] LustreError: 99115:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2871.804839] LustreError: 99115:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2875.715465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2876.176832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2880.964466] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 2888.871536] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2895.021578] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2901.521591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2901.905705] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2907.070631] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2908.232956] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 2916.913218] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2927.587488] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2927.596314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2927.603143] Lustre: Skipped 1 previous similar message [ 2931.740453] Lustre: server umount lustre-OST0000 complete [ 2937.833580] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2937.841922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2937.846825] Lustre: Skipped 1 previous similar message [ 2937.851094] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2941.943806] Lustre: server umount lustre-MDT0000 complete [ 2942.944356] LustreError: 100357:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2942.986536] LustreError: 100357:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2945.958232] LustreError: 99096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788667290 with bad export cookie 3388461600336420294 [ 2945.965550] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2945.973813] LustreError: 99096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2946.313216] Lustre: server umount lustre-MDT0001 complete [ 2957.492479] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 2960.158051] Key type lgssc unregistered [ 2960.426786] LNet: 102662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2960.449586] LNetError: 102662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2960.480287] LNet: Removed LNI 192.168.202.125@tcp [ 2961.369787] Key type .llcrypt unregistered [ 2961.371984] Key type ._llcrypt unregistered [ 2975.616687] Key type ._llcrypt registered [ 2975.618488] Key type .llcrypt registered [ 2975.727503] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 2976.839202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2976.889083] alg: No test for adler32 (adler32-zlib) [ 2978.066306] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 2978.383884] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 2980.111197] Key type lgssc registered [ 2981.678352] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2989.021637] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2989.031837] LustreError: 103735:0:(tgt_mount.c:2575:server_fill_super()) cfs_race id 716 sleeping [ 2989.059311] LustreError: 103738:0:(tgt_mount.c:2575:server_fill_super()) cfs_fail_race id 716 waking [ 2989.063908] LustreError: 103735:0:(tgt_mount.c:2575:server_fill_super()) cfs_fail_race id 716 awake: rc=4977 [ 2989.068146] LustreError: 103735:0:(tgt_mount.c:2587:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2989.080436] LustreError: 103735:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2989.087571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2990.540544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2997.669524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2997.990570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3001.918286] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 3006.007558] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing lsmod [ 3008.660131] LustreError: 104933:0:(tgt_mount.c:2575:server_fill_super()) cfs_race id 716 sleeping [ 3008.720389] LustreError: 104944:0:(tgt_mount.c:2575:server_fill_super()) cfs_fail_race id 716 waking [ 3008.726775] LustreError: 104933:0:(tgt_mount.c:2575:server_fill_super()) cfs_fail_race id 716 awake: rc=4940 [ 3008.736294] LustreError: 104933:0:(tgt_mount.c:2587:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3008.745237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3008.747108] LustreError: 104933:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3009.172936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3011.890720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3011.902883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3011.905387] Lustre: Skipped 1 previous similar message [ 3011.929366] Lustre: Skipped 2 previous similar messages [ 3017.187161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3018.161468] Lustre: server umount lustre-MDT0000 complete [ 3021.052610] LustreError: 103753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788667366 with bad export cookie 4712079167196019854 [ 3021.065783] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3021.428097] Lustre: server umount lustre-MDT0001 complete [ 3034.223882] Lustre: server umount lustre-OST0000 complete [ 3042.881072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3043.445356] LustreError: 106007:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3043.463336] LustreError: 106007:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3043.539517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3048.011703] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 3048.935188] LustreError: 106008:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3054.050411] LustreError: 106008:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3056.019280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3056.357694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3060.287219] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 3067.354993] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3072.914246] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3078.978435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3079.401399] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3085.550280] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 3089.659140] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:65) [ 3092.123783] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3100.128975] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3100.141139] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3100.151454] Lustre: Skipped 1 previous similar message [ 3100.170640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3100.174171] Lustre: Skipped 2 previous similar messages [ 3105.254727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3105.262348] Lustre: Skipped 2 previous similar messages [ 3105.980438] Lustre: server umount lustre-OST0000 complete [ 3110.372342] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3110.375377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3110.386818] Lustre: Skipped 2 previous similar messages [ 3110.399231] Lustre: Skipped 1 previous similar message [ 3120.613908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3120.630356] Lustre: Skipped 3 previous similar messages [ 3124.191129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3124.332577] Lustre: server umount lustre-MDT0000 complete [ 3125.730168] LustreError: 107847:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3125.753814] LustreError: 107847:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3128.058062] LustreError: 105987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788667473 with bad export cookie 4712079167196020547 [ 3128.061316] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3128.063169] LustreError: 105987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3128.391117] Lustre: server umount lustre-MDT0001 complete [ 3136.990347] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 3139.296847] Key type lgssc unregistered [ 3139.577705] LNet: 109540:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3139.590259] LNetError: 109540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3139.610816] LNet: Removed LNI 192.168.202.125@tcp [ 3140.369204] Key type .llcrypt unregistered [ 3140.372215] Key type ._llcrypt unregistered [ 3170.354871] Key type ._llcrypt registered [ 3170.356113] Key type .llcrypt registered [ 3170.524638] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 3171.900979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3171.922281] alg: No test for adler32 (adler32-zlib) [ 3173.083952] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 3173.305298] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 3175.008168] Key type lgssc registered [ 3176.398245] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3183.701318] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3183.713138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3185.186908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3188.990576] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 3195.367143] LustreError: 110965:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3195.387482] LustreError: 110965:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3197.698924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3198.139831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3202.283760] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 3209.124922] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3214.563252] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3221.249776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3221.672412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3229.888660] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 3230.934283] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3236.430889] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3241.186242] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3244.360924] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3248.610097] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3248.615654] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3248.635623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3249.992719] Lustre: server umount lustre-OST0000 complete [ 3254.752123] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3254.765509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3254.778849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3256.800799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3256.802442] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3261.921915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3261.939577] Lustre: Skipped 2 previous similar messages [ 3264.075922] Lustre: server umount lustre-MDT0000 complete [ 3267.040724] LustreError: 110965:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3267.071032] LustreError: 110965:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3268.076769] LustreError: 110944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788667613 with bad export cookie 10496061229271340301 [ 3268.081705] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3268.088214] LustreError: 110944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3268.504761] Lustre: server umount lustre-MDT0001 complete [ 3280.630352] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 3283.688884] Key type lgssc unregistered [ 3283.953125] LNet: 114600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3283.961241] LNetError: 114600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3285.031428] LNet: Removed LNI 192.168.202.125@tcp [ 3285.864782] Key type .llcrypt unregistered [ 3285.867660] Key type ._llcrypt unregistered [ 3303.202619] Key type ._llcrypt registered [ 3303.204561] Key type .llcrypt registered [ 3303.282176] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 3304.287167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3304.339795] alg: No test for adler32 (adler32-zlib) [ 3305.670988] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 3306.086953] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 3307.895323] Key type lgssc registered [ 3309.077289] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3318.278268] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3318.313514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3319.855207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3325.693435] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 3330.018166] LustreError: 115820:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3330.027934] LustreError: 115820:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3335.143186] LustreError: 115819:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3335.202530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3335.526882] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3339.896642] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 3347.761245] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3354.302062] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3363.602263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3364.203893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3365.252906] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3371.103444] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 3379.408098] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3390.951643] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3390.959128] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3390.961749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3390.976294] Lustre: Skipped 1 previous similar message [ 3391.004851] LustreError: Skipped 1 previous similar message [ 3395.833323] Lustre: server umount lustre-OST0000 complete [ 3401.186826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3401.211594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3401.212256] Lustre: Skipped 1 previous similar message [ 3401.223204] Lustre: Skipped 2 previous similar messages [ 3406.316317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3406.344687] Lustre: Skipped 1 previous similar message [ 3406.561118] Lustre: server umount lustre-MDT0000 complete [ 3411.235043] LustreError: 115802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788667756 with bad export cookie 12301705159820649394 [ 3411.247430] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3411.612585] Lustre: server umount lustre-MDT0001 complete [ 3423.468816] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 3426.719386] Key type lgssc unregistered [ 3427.017489] LNet: 119353:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3427.030822] LNetError: 119353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3427.080253] LNet: Removed LNI 192.168.202.125@tcp [ 3428.505473] Key type .llcrypt unregistered [ 3428.516865] Key type ._llcrypt unregistered [ 3456.054996] Key type ._llcrypt registered [ 3456.058155] Key type .llcrypt registered [ 3456.214922] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 3457.265639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3457.294278] alg: No test for adler32 (adler32-zlib) [ 3458.646822] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 3458.975384] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 3460.751211] Key type lgssc registered [ 3462.247401] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3470.389156] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3470.411508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3471.824748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3476.773324] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 3481.065208] LustreError: 120767:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3481.088442] LustreError: 120767:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3485.152200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3485.555495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3490.277316] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 3499.379756] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3506.592196] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3515.724146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3516.478243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3521.539024] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 3523.863509] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 3530.649793] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3537.836271] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3542.677788] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3542.687516] Lustre: Skipped 2 previous similar messages [ 3546.725440] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3546.769329] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3547.822930] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3547.827563] Lustre: Skipped 2 previous similar messages [ 3556.421688] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3556.425736] Lustre: Skipped 1 previous similar message [ 3562.279403] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.25@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3562.294220] Lustre: Skipped 2 previous similar messages [ 3565.173588] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.25@tcp 192.168.0.[2,10]@tcp [ 3576.635756] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.25/24@tcp in log lustre-MDT0000 [ 3576.645845] Lustre: Skipped 2 previous similar messages [ 3586.098810] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.25/24@tcp [ 3586.105428] Lustre: Skipped 1 previous similar message [ 3592.163742] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3592.166402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3592.183396] Lustre: Skipped 1 previous similar message [ 3592.194785] Lustre: Skipped 1 previous similar message [ 3597.293615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3597.305409] Lustre: Skipped 1 previous similar message [ 3598.240937] Lustre: server umount lustre-OST0000 complete [ 3603.425896] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3603.432593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3603.446284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3607.523132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3607.524851] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3609.784763] Lustre: server umount lustre-MDT0000 complete [ 3612.645253] LustreError: 120760:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3612.684119] LustreError: 120760:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3614.244563] LustreError: 120747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788667959 with bad export cookie 18333275974866993369 [ 3614.245798] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3614.255702] LustreError: 120747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3614.943609] Lustre: server umount lustre-MDT0001 complete [ 3628.686308] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 3632.111147] Key type lgssc unregistered [ 3632.377715] LNet: 125811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3632.384916] LNetError: 125811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3633.445768] LNet: Removed LNI 192.168.202.125@tcp [ 3634.477893] Key type .llcrypt unregistered [ 3634.479861] Key type ._llcrypt unregistered [ 3666.038655] Key type ._llcrypt registered [ 3666.040856] Key type .llcrypt registered [ 3666.176088] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 3667.513139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3667.654550] alg: No test for adler32 (adler32-zlib) [ 3668.771565] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 3669.073392] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 3670.775254] Key type lgssc registered [ 3672.077782] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3682.142465] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3682.167706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3683.834488] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3683.874024] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.25/24@tcp [ 3683.899683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3689.139670] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 3694.051949] LustreError: 127373:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3694.096050] LustreError: 127373:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3699.176239] LustreError: 127374:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3699.208969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3699.585480] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3699.615243] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.25/24@tcp [ 3699.700913] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3706.028195] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 3715.256217] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3723.629601] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3731.436229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3731.827534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3738.604280] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 3746.242712] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3754.365717] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:169 to 0x280000401:193) [ 3769.835612] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3769.838485] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3769.850851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3769.862048] LustreError: Skipped 1 previous similar message [ 3769.865080] Lustre: Skipped 1 previous similar message [ 3771.909382] Lustre: server umount lustre-OST0000 complete [ 3780.070930] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3780.073553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3780.086245] Lustre: Skipped 1 previous similar message [ 3780.093134] Lustre: Skipped 1 previous similar message [ 3783.761368] Lustre: server umount lustre-MDT0000 complete [ 3785.184140] LustreError: 127374:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3785.198367] LustreError: 127374:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3788.395541] LustreError: 127352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788668133 with bad export cookie 2660928180676104988 [ 3788.409977] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3788.413701] LustreError: 127352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3788.774678] Lustre: server umount lustre-MDT0001 complete [ 3800.501571] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 3803.808878] Key type lgssc unregistered [ 3804.113271] LNet: 130987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3804.123688] LNetError: 130987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3804.139325] LNet: Removed LNI 192.168.202.125@tcp [ 3805.284087] Key type .llcrypt unregistered [ 3805.289663] Key type ._llcrypt unregistered [ 3837.865352] Key type ._llcrypt registered [ 3837.872523] Key type .llcrypt registered [ 3838.110880] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_hostid [ 3861.838775] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 3863.173270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3863.231310] alg: No test for adler32 (adler32-zlib) [ 3864.352984] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 3864.607530] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 3866.305047] Key type lgssc registered [ 3867.573134] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3909.003982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3909.034694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3910.274709] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3910.299996] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3910.376600] Lustre: lustre-MDT0000: new disk, initializing [ 3910.464056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3910.480448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3916.227337] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 3928.925406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3929.179609] Lustre: 134396:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3929.404552] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3929.410597] Lustre: Skipped 1 previous similar message [ 3929.573287] Lustre: lustre-MDT0001: new disk, initializing [ 3929.788230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3929.878350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3929.908552] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3937.889418] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 3953.463513] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3960.497754] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3969.185463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3969.542263] Lustre: lustre-OST0000: new disk, initializing [ 3969.555489] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3969.572531] Lustre: 136367:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3969.730611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3977.778565] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 3979.955297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3979.978202] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3980.310232] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3988.682779] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3994.157282] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3994.348738] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3999.267526] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3999.539706] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4005.867412] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4005.872804] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4005.875427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4005.886800] Lustre: Skipped 1 previous similar message [ 4005.905499] Lustre: Skipped 1 previous similar message [ 4010.985826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4011.006861] Lustre: Skipped 1 previous similar message [ 4016.096405] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4016.125659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4016.143047] Lustre: Skipped 1 previous similar message [ 4016.411761] Lustre: server umount lustre-OST0000 complete [ 4021.220214] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4021.227723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4021.230480] Lustre: Skipped 1 previous similar message [ 4026.340173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4026.349150] Lustre: Skipped 2 previous similar messages [ 4027.498984] Lustre: server umount lustre-MDT0000 complete [ 4033.078539] LustreError: 134389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788668378 with bad export cookie 3857595367164666245 [ 4033.087435] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4033.105983] LustreError: 134389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4033.648535] Lustre: server umount lustre-MDT0001 complete [ 4043.230838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4043.839239] LustreError: 138586:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4043.866923] LustreError: 138586:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4043.992909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4049.379213] LustreError: 138586:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4049.946912] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 4053.474143] LustreError: 138587:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4058.601488] LustreError: 138586:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4058.856925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4059.273487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4064.752646] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 4073.322559] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4079.883098] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4087.174939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4087.567415] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4094.306085] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 4102.004860] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4107.910440] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4108.224307] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4112.768873] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4112.998985] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4122.044382] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4122.378536] Lustre: lustre-OST0001: new disk, initializing [ 4122.385207] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4122.409656] Lustre: 141957:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4122.542035] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4129.368542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4129.384440] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4129.453302] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 4129.511634] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4137.630430] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4137.822747] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4142.247737] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4142.524051] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4149.254267] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9336d16c8800.ost_server_uuid 50 [ 4150.964733] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9336d16c8800.ost_server_uuid in IDLE state after 0 sec [ 4160.485720] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4160.517537] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4160.531107] LustreError: Skipped 1 previous similar message [ 4160.533880] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4160.550839] Lustre: Skipped 1 previous similar message [ 4161.797855] Lustre: server umount lustre-OST0001 complete [ 4165.611825] LustreError: 140408:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4165.639808] LustreError: 140408:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4166.625481] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4166.635734] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4166.674270] Lustre: Skipped 1 previous similar message [ 4170.730799] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4170.736698] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4172.289494] Lustre: server umount lustre-OST0000 complete [ 4176.870219] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4176.877122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4176.882681] Lustre: Skipped 3 previous similar messages [ 4180.961329] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4180.974348] Lustre: Skipped 1 previous similar message [ 4182.351315] Lustre: server umount lustre-MDT0000 complete [ 4186.083974] LustreError: 138586:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4186.106763] LustreError: 138586:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4186.224631] LustreError: 138567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788668531 with bad export cookie 3857595367164667127 [ 4186.237446] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4186.247993] LustreError: 138567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4186.687723] Lustre: server umount lustre-MDT0001 complete [ 4208.102220] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 4211.093281] Key type lgssc unregistered [ 4211.455952] LNet: 144957:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4211.468402] LNetError: 144957:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4211.504198] LNet: Removed LNI 192.168.202.125@tcp [ 4212.487658] Key type .llcrypt unregistered [ 4212.490516] Key type ._llcrypt unregistered [ 4255.583836] Key type ._llcrypt registered [ 4255.585584] Key type .llcrypt registered [ 4255.706651] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_hostid [ 4271.782133] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 4272.998392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4273.041394] alg: No test for adler32 (adler32-zlib) [ 4274.355696] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 4274.688888] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 4276.423714] Key type lgssc registered [ 4277.964184] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4314.180431] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4314.213300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4315.546762] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4315.599244] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4315.684777] Lustre: lustre-MDT0000: new disk, initializing [ 4315.788654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4315.816546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4321.011941] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 4332.137345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4332.275207] Lustre: 149112:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4332.329226] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4332.333424] Lustre: Skipped 1 previous similar message [ 4332.423664] Lustre: lustre-MDT0001: new disk, initializing [ 4332.536874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4332.564146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4332.578818] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4337.778701] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 4348.315957] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4354.298509] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4361.180585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4361.529492] Lustre: lustre-OST0000: new disk, initializing [ 4361.533196] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4361.539268] Lustre: 151080:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4361.599567] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4367.436403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4367.458272] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4367.656033] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4368.498516] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 4379.187116] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4383.500599] Lustre: Failing over lustre-OST0000 [ 4383.847167] Lustre: server umount lustre-OST0000 complete [ 4401.631111] Lustre: 147333:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788668730/real 1788668730] req@000000004a4b06fa x1875554985872128/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788668746 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4401.645871] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4402.231829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4402.494259] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4402.528409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4402.539435] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4404.575138] Lustre: 147333:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788668733/real 1788668733] req@000000005387d21c x1875554985873152/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788668749 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4406.574294] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4408.494541] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 4409.824680] Lustre: 147335:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788668738/real 1788668738] req@0000000086f9ea20 x1875554985874432/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788668754 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4409.850386] Lustre: 147335:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4411.836066] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4411.840886] Lustre: lustre-OST0000: Recovery over after 0:05, of 3 clients 3 recovered and 0 were evicted. [ 4412.222092] Lustre: Failing over lustre-MDT0000 [ 4412.491684] Lustre: server umount lustre-MDT0000 complete [ 4414.303153] Lustre: 147334:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788668743/real 1788668743] req@00000000c948b70e x1875554985875456/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788668759 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4414.321984] Lustre: 147334:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4414.434166] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4414.441300] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4417.002815] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4417.023671] Lustre: Skipped 1 previous similar message [ 4417.783581] LustreError: 149117:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4417.809093] LustreError: 149117:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4420.065090] LustreError: 149118:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4420.083602] LustreError: 149118:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4422.930222] LustreError: 150210:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4425.194548] LustreError: 149123:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4425.212855] LustreError: 149123:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4430.311381] LustreError: 149122:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4430.342084] LustreError: 149122:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4431.465235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4431.555436] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4431.869985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4431.924425] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4433.145204] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4436.297060] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 4436.981302] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4436.993554] Lustre: Skipped 1 previous similar message [ 4437.022920] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4437.075581] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4447.205559] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4447.207506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4447.249755] Lustre: Skipped 1 previous similar message [ 4447.260091] Lustre: Skipped 1 previous similar message [ 4449.918466] Lustre: server umount lustre-OST0000 complete [ 4457.441761] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4457.442238] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4457.470456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4457.482129] Lustre: Skipped 1 previous similar message [ 4459.932575] Lustre: server umount lustre-MDT0000 complete [ 4462.548610] LustreError: 151091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788668807 with bad export cookie 9418163533361258593 [ 4462.552351] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4462.557904] LustreError: 151091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4462.757742] Lustre: server umount lustre-MDT0001 complete [ 4472.257422] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 4474.623781] Key type lgssc unregistered [ 4474.923780] LNet: 154731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4474.939776] LNetError: 154731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4474.951232] LNet: Removed LNI 192.168.202.125@tcp [ 4475.861868] Key type .llcrypt unregistered [ 4475.866767] Key type ._llcrypt unregistered [ 4511.324941] Key type ._llcrypt registered [ 4511.327198] Key type .llcrypt registered [ 4511.393362] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 4512.322802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4512.418714] alg: No test for adler32 (adler32-zlib) [ 4513.411899] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 4513.653322] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 4515.303178] Key type lgssc registered [ 4516.258661] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4523.319158] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4523.345744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4524.816214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4529.110821] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 4534.245818] LustreError: 156248:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4534.270587] LustreError: 156248:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4537.840476] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4539.361981] LustreError: 156249:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4541.799810] LustreError: 156277:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4541.807463] LustreError: 156277:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4541.814275] LustreError: 156277:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 4542.218770] Lustre: server umount lustre-MDT0000 complete [ 4555.368890] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_hostid [ 4565.942155] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 4604.858369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4605.100281] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4605.131993] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4605.243875] Lustre: lustre-MDT0000: new disk, initializing [ 4605.364463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4605.384747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4609.399873] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 4619.134130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4619.247394] Lustre: 159582:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4619.295686] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4619.298274] Lustre: Skipped 1 previous similar message [ 4619.347720] Lustre: lustre-MDT0001: new disk, initializing [ 4619.460383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4619.493342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4619.511618] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4623.906724] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 4632.964714] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4637.749830] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4643.861905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4644.159179] Lustre: lustre-OST0000: new disk, initializing [ 4644.163309] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4644.169289] Lustre: 161550:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4644.275349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4650.404552] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 4650.556057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4650.565058] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4650.754866] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4659.555947] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4663.401931] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4665.234298] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 00:30:09 (1788669009) [ 4891.111231] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4891.114429] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4891.137768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4891.622239] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4891.648062] Lustre: Skipped 1 previous similar message [ 4891.666723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4891.674836] Lustre: Skipped 1 previous similar message [ 4895.732439] Lustre: server umount lustre-MDT0000 complete [ 4896.738734] LustreError: 163304:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4896.770478] LustreError: 163304:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4900.010771] LustreError: 159575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788669244 with bad export cookie 1357286239147299335 [ 4900.018254] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4900.028982] LustreError: 159575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4901.856268] LustreError: 159590:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4901.856421] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4901.918051] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4905.888760] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4905.888760] LustreError: 163319:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4906.385579] Lustre: server umount lustre-MDT0001 complete [ 4916.605710] Lustre: server umount lustre-OST0000 complete [ 4928.158877] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_hostid [ 4939.550415] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 4978.407615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4978.592200] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4978.622287] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4978.713983] Lustre: lustre-MDT0000: new disk, initializing [ 4978.878297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4978.899400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4983.262559] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 4993.288524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4993.489835] Lustre: 166209:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4993.597590] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4993.604153] Lustre: Skipped 1 previous similar message [ 4993.700897] Lustre: lustre-MDT0001: new disk, initializing [ 4993.857560] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4993.898371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4993.909823] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4998.260752] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 5007.684365] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5014.544043] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5021.775597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5022.030126] Lustre: lustre-OST0000: new disk, initializing [ 5022.033580] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5022.039972] Lustre: 168176:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5022.122802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5028.602503] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 5032.050523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5032.055688] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5032.174693] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5038.766749] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5044.311453] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5044.659660] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5049.703302] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5050.058906] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5052.911222] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5052.912636] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5052.962402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5052.976573] Lustre: Skipped 1 previous similar message [ 5063.142814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5063.152741] Lustre: Skipped 2 previous similar messages [ 5066.207126] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5066.670323] Lustre: server umount lustre-OST0000 complete [ 5073.379945] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5073.406198] Lustre: Skipped 2 previous similar messages [ 5077.858751] Lustre: server umount lustre-MDT0000 complete [ 5078.498700] LustreError: 166221:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5078.517606] LustreError: 166221:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5081.731266] LustreError: 166202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788669426 with bad export cookie 1357286239147458165 [ 5081.749537] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5082.061619] Lustre: server umount lustre-MDT0001 complete [ 5090.335513] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 00:37:14 (1788669434) [ 5106.678901] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_hostid [ 5116.612208] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 5153.390955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5153.805592] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5153.889949] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5154.002641] Lustre: lustre-MDT0000: new disk, initializing [ 5154.117486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5154.142190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5158.643844] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 5170.359828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5170.574742] Lustre: 172936:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5171.161360] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5176.823926] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 5187.105220] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5193.225587] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5200.202390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5200.510430] Lustre: lustre-OST0000: new disk, initializing [ 5200.513396] Lustre: Skipped 1 previous similar message [ 5200.517679] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5200.523794] Lustre: Skipped 2 previous similar messages [ 5200.530796] Lustre: 174911:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5205.940758] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 5210.732704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5210.752851] Lustre: Skipped 1 previous similar message [ 5210.759125] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5210.856430] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5215.481375] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5226.471202] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5226.475845] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5226.491731] LustreError: Skipped 1 previous similar message [ 5226.497610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5226.497619] Lustre: Skipped 2 previous similar messages [ 5226.537471] Lustre: Skipped 1 previous similar message [ 5227.534842] Lustre: server umount lustre-OST0000 complete [ 5232.608210] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5232.616665] LustreError: Skipped 1 previous similar message [ 5238.061302] Lustre: server umount lustre-MDT0000 complete [ 5241.833384] LustreError: 172947:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5241.857093] LustreError: 172947:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5242.278371] LustreError: 172929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788669587 with bad export cookie 1357286239147459047 [ 5242.292667] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5242.302699] LustreError: 172929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 5242.661643] Lustre: server umount lustre-MDT0001 complete [ 5251.726300] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 00:39:56 (1788669596) [ 5268.779437] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_hostid [ 5279.842769] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 5319.262702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5319.466934] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5319.508535] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5319.595530] Lustre: lustre-MDT0000: new disk, initializing [ 5319.760233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5319.763080] Lustre: Skipped 2 previous similar messages [ 5319.789222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5324.925300] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 5336.099292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5336.260978] Lustre: 179235:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5336.600495] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5341.694079] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 5352.200407] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5358.414914] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5366.569038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5366.933271] Lustre: 181211:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5372.586858] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5372.773883] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5373.491978] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 5384.644334] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5398.503937] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5398.522914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5398.524407] Lustre: Skipped 3 previous similar messages [ 5398.540304] Lustre: Skipped 5 previous similar messages [ 5400.133859] Lustre: server umount lustre-OST0000 complete [ 5408.234560] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5410.242625] Lustre: server umount lustre-MDT0000 complete [ 5413.863178] LustreError: 179243:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5413.882279] LustreError: 179243:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5415.450780] LustreError: 180044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788669760 with bad export cookie 1357286239147460132 [ 5415.453879] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5415.469369] LustreError: 180044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5415.828482] Lustre: server umount lustre-MDT0001 complete [ 5428.562668] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 5431.972491] Key type lgssc unregistered [ 5432.348321] LNet: 183070:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5432.354980] LNetError: 183070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5432.368723] LNet: Removed LNI 192.168.202.125@tcp [ 5433.173757] Key type .llcrypt unregistered [ 5433.178347] Key type ._llcrypt unregistered [ 5459.814261] Key type ._llcrypt registered [ 5459.815761] Key type .llcrypt registered [ 5459.960908] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 5461.648188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5461.703687] alg: No test for adler32 (adler32-zlib) [ 5462.865152] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 5463.139705] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 5464.852236] Key type lgssc registered [ 5466.429454] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5475.480549] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5475.514413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5477.184838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5482.436487] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 5487.594037] LustreError: 184483:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5487.628780] LustreError: 184483:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5492.400932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5492.816732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5498.394205] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 5507.523314] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5513.647860] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5521.989075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5522.466777] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5527.572573] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5530.795650] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 5539.691362] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5558.243576] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5558.246464] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5558.255309] Lustre: Skipped 1 previous similar message [ 5558.256104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5558.272460] LustreError: Skipped 1 previous similar message [ 5563.371890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5563.376545] Lustre: Skipped 2 previous similar messages [ 5564.013557] Lustre: server umount lustre-OST0000 complete [ 5568.487950] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5568.491227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5568.517628] Lustre: Skipped 1 previous similar message [ 5568.533980] Lustre: Skipped 1 previous similar message [ 5573.603396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5573.613425] Lustre: Skipped 1 previous similar message [ 5574.920255] Lustre: server umount lustre-MDT0000 complete [ 5578.720818] LustreError: 185931:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5578.749701] LustreError: 185931:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5579.587293] LustreError: 184465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788669924 with bad export cookie 1334253777305291263 [ 5579.609046] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5580.218981] Lustre: server umount lustre-MDT0001 complete [ 5593.508690] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 5596.682898] Key type lgssc unregistered [ 5596.929626] LNet: 188037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5596.949203] LNetError: 188037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5596.965921] LNet: Removed LNI 192.168.202.125@tcp [ 5598.001114] Key type .llcrypt unregistered [ 5598.006160] Key type ._llcrypt unregistered [ 5626.889241] Key type ._llcrypt registered [ 5626.891867] Key type .llcrypt registered [ 5627.034626] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 5628.515438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5628.555765] alg: No test for adler32 (adler32-zlib) [ 5629.674900] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 5630.003332] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 5631.723612] Key type lgssc registered [ 5633.030802] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5642.096443] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5642.120472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5643.488347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5648.200778] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 5653.985774] LustreError: 189452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5654.000402] LustreError: 189452:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5656.590834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5656.944420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5661.762214] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 5670.630986] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5676.367813] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5683.549700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5683.912564] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5689.017537] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5691.055531] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 5699.712989] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5706.545805] Lustre: lustre-OST0000: Not available for connect from 192.168.202.25@tcp (stopping) [ 5706.726323] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5706.732525] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5706.759244] Lustre: Skipped 1 previous similar message [ 5707.636438] Lustre: server umount lustre-OST0000 complete [ 5716.342748] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9336c7c7b000.ost_server_uuid 50 [ 5718.329529] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5733.857444] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5733.877130] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5733.891787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5733.898090] Lustre: Skipped 2 previous similar messages [ 5736.420704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5736.421240] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5737.853726] Lustre: server umount lustre-MDT0000 complete [ 5741.542241] LustreError: 189446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5741.555992] LustreError: 189446:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5742.047981] LustreError: 189433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788670087 with bad export cookie 17387124213324426736 [ 5742.079899] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5742.449846] Lustre: server umount lustre-MDT0001 complete [ 5751.596348] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 00:48:16 (1788670096) [ 5759.107462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5759.781837] LustreError: 193363:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5759.890971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5765.101722] LustreError: 193364:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5765.537379] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 5769.201111] LustreError: 193363:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5775.369117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5775.936578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5780.749158] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 5788.122598] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5793.961446] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5801.355359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5801.734625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5805.803767] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5808.344777] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 5816.056629] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5822.699166] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5822.943867] Lustre: lustre-OST0001: new disk, initializing [ 5822.947350] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5822.953422] Lustre: 196205:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5823.072173] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5829.148229] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 5832.769038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5832.773764] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5832.911951] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5839.068348] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5841.889143] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5841.895810] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5841.907256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5841.912532] Lustre: Skipped 1 previous similar message [ 5847.923568] Lustre: server umount lustre-OST0000 complete [ 5848.558946] LustreError: 195187:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5848.591027] LustreError: 195187:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5855.276833] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5855.578220] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5860.632664] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5860.852548] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5865.720405] LustreError: 195187:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5865.747717] LustreError: 195187:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 5874.146826] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5874.161889] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5874.174592] Lustre: Skipped 1 previous similar message [ 5874.188032] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5874.196107] Lustre: Skipped 4 previous similar messages [ 5880.259660] Lustre: server umount lustre-OST0001 complete [ 5884.384996] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5884.406657] Lustre: Skipped 1 previous similar message [ 5884.409900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5884.411780] Lustre: Skipped 2 previous similar messages [ 5898.207160] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5898.421241] Lustre: server umount lustre-MDT0000 complete [ 5899.750646] LustreError: 193358:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5899.780599] LustreError: 193358:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 5902.049261] LustreError: 193345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788670247 with bad export cookie 17387124213324427807 [ 5902.071927] LustreError: 193345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5902.072601] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5902.479217] Lustre: server umount lustre-MDT0001 complete [ 5927.066938] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 00:51:11 (1788670271) [ 5934.241554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5934.648871] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5934.691436] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5934.857279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5940.451439] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 5950.001539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5950.129930] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5950.155486] Lustre: 199647:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5954.752574] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 5962.025967] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5967.199576] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5974.438980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5974.599137] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5980.475878] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 5981.891554] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5987.277311] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5994.104880] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5994.229675] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6001.316684] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6001.696565] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 6010.078989] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6012.896734] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6012.905263] LustreError: Skipped 1 previous similar message [ 6012.909454] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6012.917911] Lustre: Skipped 1 previous similar message [ 6012.924314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6012.935328] Lustre: Skipped 5 previous similar messages [ 6019.067306] Lustre: server umount lustre-OST0000 complete [ 6020.876404] LustreError: 201497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6036.960414] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6042.183348] Lustre: server umount lustre-OST0001 complete [ 6047.715259] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6047.729708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6047.743662] Lustre: Skipped 4 previous similar messages [ 6047.766019] Lustre: Skipped 7 previous similar messages [ 6051.815085] Lustre: server umount lustre-MDT0000 complete [ 6054.851443] LustreError: 201518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788670399 with bad export cookie 17387124213324429214 [ 6054.858139] LustreError: 201518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6054.859237] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6055.252909] Lustre: server umount lustre-MDT0001 complete [ 6082.126483] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 00:53:47 (1788670427) [ 6098.620071] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_hostid [ 6109.227219] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 6145.942976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6146.223509] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6146.281079] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6146.386365] Lustre: lustre-MDT0000: new disk, initializing [ 6146.475314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6146.480883] Lustre: Skipped 3 previous similar messages [ 6146.492336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6151.583340] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 6163.063841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6163.211898] Lustre: 207741:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6163.256504] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6163.259986] Lustre: Skipped 1 previous similar message [ 6163.315289] Lustre: lustre-MDT0001: new disk, initializing [ 6163.427897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6163.438459] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6167.935581] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 6177.821569] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6183.180443] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6189.126995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6189.349112] Lustre: lustre-OST0000: new disk, initializing [ 6189.351726] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6189.356062] Lustre: 209712:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6196.313824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6196.334644] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6196.415625] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 6196.465562] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6207.491360] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6211.326677] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6211.425400] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6215.428318] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6215.543855] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6222.311834] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6222.338297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6222.348034] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6222.365025] LustreError: Skipped 2 previous similar messages [ 6223.524490] Lustre: server umount lustre-OST0000 complete [ 6233.851822] Lustre: server umount lustre-MDT0000 complete [ 6237.592128] LustreError: 207734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788670582 with bad export cookie 17387124213324430999 [ 6237.598638] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6237.604694] LustreError: 207734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6237.939297] Lustre: server umount lustre-MDT0001 complete [ 6246.419579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6246.915405] LustreError: 211930:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6246.942415] LustreError: 211930:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 6252.261861] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 6261.589349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6267.199507] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 6274.824417] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6281.039408] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6288.158961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6288.608665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6288.611602] Lustre: Skipped 4 previous similar messages [ 6295.861484] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 6302.586980] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6307.609902] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6307.856026] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6312.983132] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6313.170420] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6316.517254] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6316.526800] LustreError: Skipped 1 previous similar message [ 6329.311480] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6329.462702] Lustre: server umount lustre-OST0000 complete [ 6336.174919] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6339.188760] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6369.377198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6377.430994] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 6382.016285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6382.039222] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6382.040938] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6382.078213] Lustre: Skipped 1 previous similar message [ 6386.730671] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6388.720188] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6392.289746] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6392.297783] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6392.312428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6392.319219] Lustre: Skipped 6 previous similar messages [ 6392.333739] Lustre: Skipped 11 previous similar messages [ 6398.014719] Lustre: server umount lustre-OST0000 complete [ 6408.315336] Lustre: server umount lustre-MDT0000 complete [ 6412.647884] LustreError: 211911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788670757 with bad export cookie 17387124213324431874 [ 6412.660348] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6412.661135] LustreError: 211911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6412.980537] Lustre: server umount lustre-MDT0001 complete [ 6422.097818] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 00:59:26 (1788670766) [ 6428.822837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6434.759542] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 6443.870333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6448.780427] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 6457.072144] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6462.928533] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6469.985321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6476.586788] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 6486.891337] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6491.992844] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6492.159109] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6497.722948] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6498.109689] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6505.930633] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6506.205334] Lustre: lustre-OST0001: new disk, initializing [ 6506.212082] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6506.220786] Lustre: 221270:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6511.323665] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 6515.849680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6515.863150] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6515.974046] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6520.417754] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6525.788349] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6526.063053] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6530.796902] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6530.994877] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6536.673628] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6538.914886] Lustre: server umount lustre-OST0001 complete [ 6541.794700] LustreError: 219720:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6541.822103] LustreError: 219720:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 6547.256453] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6547.512403] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6553.553571] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6553.857767] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6583.624145] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6583.929961] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6583.936402] Lustre: Skipped 5 previous similar messages [ 6589.436199] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6589.459316] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6589.459771] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6590.975498] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 6599.333081] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6601.315975] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6609.648782] Lustre: server umount lustre-OST0001 complete [ 6633.483298] Lustre: server umount lustre-OST0000 complete [ 6644.352221] Lustre: server umount lustre-MDT0000 complete [ 6649.073691] LustreError: 218634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788670994 with bad export cookie 17387124213324432847 [ 6649.080322] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6655.584978] Lustre: server umount lustre-MDT0001 complete [ 6686.782638] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 01:03:51 (1788671031) [ 6694.601844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6694.928742] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6695.003975] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6700.459489] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 6710.333093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6710.519429] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6710.545902] Lustre: 226525:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6715.923887] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 6723.071520] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6729.284413] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6736.993460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6737.164537] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6743.772620] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 6753.275620] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6763.895916] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6763.990534] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6771.323969] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 6779.661737] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6781.144264] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6786.008712] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6786.205694] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6791.842538] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6792.129982] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6799.217602] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9336c2bb5000.ost_server_uuid 50 [ 6801.205619] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9336c2bb5000.ost_server_uuid in IDLE state after 0 sec [ 6802.570968] Lustre: Permanently deactivating lustre-OST0001 [ 6802.614948] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6807.368168] Lustre: Permanently reactivating lustre-OST0001 [ 6807.389820] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6807.391799] Lustre: Skipped 2 previous similar messages [ 6813.174786] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6813.185677] LustreError: Skipped 6 previous similar messages [ 6813.189916] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6813.208295] Lustre: Skipped 10 previous similar messages [ 6813.213735] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6813.220350] Lustre: Skipped 17 previous similar messages [ 6818.858367] Lustre: server umount lustre-OST0001 complete [ 6828.773161] Lustre: server umount lustre-OST0000 complete [ 6839.385619] Lustre: server umount lustre-MDT0000 complete [ 6842.403075] LustreError: 227267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788671187 with bad export cookie 17387124213324434128 [ 6842.410457] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6842.412285] LustreError: 227267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6842.916583] Lustre: server umount lustre-MDT0001 complete [ 6870.189355] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 01:06:54 (1788671214) [ 6879.573919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6880.079281] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6880.135880] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6880.146773] Lustre: Skipped 2 previous similar messages [ 6885.297103] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 6894.916641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6895.000575] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6895.039456] Lustre: 233225:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6900.351656] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 6907.641170] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6914.257736] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6921.284168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6921.476533] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6921.495885] Lustre: Permanently deactivating lustre-OST0000 [ 6921.504841] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6928.163808] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 6928.478595] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6936.768583] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6944.525505] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6944.690701] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6951.652572] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6952.396664] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 6960.190926] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6961.790739] Lustre: Permanently reactivating lustre-OST0000 [ 6961.800179] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6961.804488] Lustre: Skipped 2 previous similar messages [ 6970.895392] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 6970.901502] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 42 seconds. I think it's dead, and I am evicting it. exp 0000000004c8791d, cur 1788671315 deadline 1788671303 last 1788671273 [ 6970.946160] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6984.381549] Lustre: server umount lustre-OST0001 complete [ 6995.520718] Lustre: server umount lustre-OST0000 complete [ 7014.367224] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7014.778954] Lustre: server umount lustre-MDT0000 complete [ 7019.401676] LustreError: 233217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788671364 with bad export cookie 17387124213324435808 [ 7019.424826] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7019.426345] LustreError: 233217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7019.683598] Lustre: server umount lustre-MDT0001 complete [ 7030.816354] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 7033.680132] Key type lgssc unregistered [ 7034.043765] LNet: 238104:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7034.055263] LNetError: 238104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7034.073223] LNet: Removed LNI 192.168.202.125@tcp [ 7034.841158] Key type .llcrypt unregistered [ 7034.843442] Key type ._llcrypt unregistered [ 7058.685425] Key type ._llcrypt registered [ 7058.688348] Key type .llcrypt registered [ 7058.817802] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 7060.854157] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7061.047970] alg: No test for adler32 (adler32-zlib) [ 7062.263662] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 7062.738959] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 7064.431210] Key type lgssc registered [ 7066.086868] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7076.785773] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7076.818149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7078.451659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7084.289888] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 7088.620153] LustreError: 239567:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7088.649853] LustreError: 239567:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7093.728917] LustreError: 239566:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7094.495222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7094.635344] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7094.659696] Lustre: 239554:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7094.675708] Lustre: Permanently deactivating lustre-MDT0001 [ 7094.689285] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7094.950890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7100.146982] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 7102.486424] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7108.809839] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7115.106748] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7122.040620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7122.358462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7123.363499] LustreError: 241412:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7123.377293] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7128.555616] LustreError: 241412:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7129.153558] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 7133.666875] LustreError: 241410:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7137.824373] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7144.937044] LustreError: 241410:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7144.959051] LustreError: 241410:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7146.364436] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7146.692072] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7152.132864] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7152.167447] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7154.984801] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 7163.254246] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7166.712315] Lustre: Permanently reactivating lustre-MDT0001 [ 7166.723641] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7166.729869] Lustre: Skipped 2 previous similar messages [ 7173.449582] Lustre: lustre-MDT0001: Client 071bda40-b1db-4a61-8c11-8100d19158a2 (at 192.168.202.25@tcp) reconnecting [ 7176.855147] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7176.892044] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7176.905557] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7176.930784] LustreError: 243233:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7176.931151] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7176.946909] LustreError: 243233:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7176.947077] LustreError: 243233:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.125@tcp: cfg command failed: rc = -108 [ 7176.947101] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7176.947101] [ 7176.947332] LustreError: 239558:0:(mgc_request.c:612:do_requeue()) failed processing log: -108 [ 7181.049144] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7182.931697] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7190.615720] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7190.834618] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7196.578417] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7196.903369] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7199.680326] Lustre: Permanently deactivating lustre-MDT0001 [ 7199.731913] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7199.739132] Lustre: Skipped 2 previous similar messages [ 7205.474416] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7217.636578] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7217.647110] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7217.677451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7218.656545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7218.656755] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7218.671032] Lustre: Skipped 2 previous similar messages [ 7218.710332] Lustre: Skipped 2 previous similar messages [ 7223.777798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7223.780402] Lustre: Skipped 3 previous similar messages [ 7228.384925] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7228.535933] Lustre: server umount lustre-MDT0000 complete [ 7228.899656] LustreError: 239561:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7228.915570] LustreError: 239561:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7233.501877] LustreError: 239546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788671578 with bad export cookie 4103568412893335850 [ 7233.503356] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7233.516082] LustreError: 239546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7234.335822] Lustre: server umount lustre-MDT0001 complete [ 7239.926503] Lustre: server umount lustre-OST0000 complete [ 7253.749983] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 01:13:18 (1788671598) [ 7264.554536] Lustre: server umount lustre-OST0001 complete [ 7272.501675] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_hostid [ 7284.386396] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 7325.982228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7326.279485] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7326.292483] Lustre: Skipped 2 previous similar messages [ 7326.318629] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7326.453228] Lustre: lustre-MDT0000: new disk, initializing [ 7326.624647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7326.649651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7332.487850] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 7344.060255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7344.256365] Lustre: 247833:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7344.313710] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7344.318028] Lustre: Skipped 1 previous similar message [ 7344.409166] Lustre: lustre-MDT0001: new disk, initializing [ 7344.521719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7344.600308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7344.616523] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7350.326264] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 7360.824847] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7367.113260] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7374.784049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7375.135511] Lustre: lustre-OST0000: new disk, initializing [ 7375.142442] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7375.146250] Lustre: 249802:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7375.258664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7383.074109] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 7384.661887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7384.671475] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7384.863874] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7393.689468] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7396.623532] LustreError: 247840:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7406.333133] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7406.561129] Lustre: lustre-OST0001: new disk, initializing [ 7406.568581] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7406.578964] Lustre: 250916:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7406.681302] LustreError: 247840:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7406.718584] LustreError: 247839:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7413.791240] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 7415.889301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7415.897146] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7415.982117] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7416.736565] LustreError: 247839:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7424.514023] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7426.663209] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7431.660295] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7431.667885] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7431.690459] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7431.696423] Lustre: Skipped 1 previous similar message [ 7435.379036] Lustre: server umount lustre-OST0001 complete [ 7436.798523] LustreError: 249788:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7436.825601] LustreError: 249788:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7456.737938] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7456.739939] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7456.760022] LustreError: Skipped 2 previous similar messages [ 7456.764964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7456.781975] Lustre: Skipped 2 previous similar messages [ 7460.103740] Lustre: server umount lustre-OST0000 complete [ 7467.488733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7467.532173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7467.540305] Lustre: Skipped 3 previous similar messages [ 7472.209430] Lustre: server umount lustre-MDT0000 complete [ 7477.604602] LustreError: 247827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788671822 with bad export cookie 4103568412893338447 [ 7477.614091] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7477.621278] LustreError: 247827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7484.054610] Lustre: server umount lustre-MDT0001 complete [ 7496.240911] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 7500.109341] Key type lgssc unregistered [ 7500.398045] LNet: 252952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7500.403207] LNetError: 252952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7500.416303] LNet: Removed LNI 192.168.202.125@tcp [ 7501.416347] Key type .llcrypt unregistered [ 7501.417519] Key type ._llcrypt unregistered [ 7549.670913] Key type ._llcrypt registered [ 7549.674236] Key type .llcrypt registered [ 7549.819943] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 7550.948556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7551.156098] alg: No test for adler32 (adler32-zlib) [ 7552.341887] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 7552.516626] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 7554.351162] Key type lgssc registered [ 7555.767733] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7566.042481] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7566.066101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7567.466187] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7567.513900] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7567.834165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7574.325534] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 7584.795646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7584.888311] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7584.956928] Lustre: 255157:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7585.424840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7590.370267] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 7598.194726] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7605.118793] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7612.942299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7613.116664] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7613.328660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7619.266649] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7620.005455] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 7627.918622] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7685.601574] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7685.612260] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7685.657046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7686.118279] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7686.133030] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7686.160244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7686.167692] Lustre: Skipped 1 previous similar message [ 7689.528779] Lustre: server umount lustre-OST0000 complete [ 7694.994340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7708.379552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7708.777048] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7711.854403] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7711.869868] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7715.316594] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 7725.239639] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7726.189902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7726.210665] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7767.526632] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7767.530200] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7767.547493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7767.560379] Lustre: Skipped 1 previous similar message [ 7767.580873] Lustre: Skipped 1 previous similar message [ 7771.448748] Lustre: server umount lustre-OST0000 complete [ 7777.761488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7777.777385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7777.788079] Lustre: Skipped 1 previous similar message [ 7782.880365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7782.887409] Lustre: Skipped 2 previous similar messages [ 7790.049590] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7790.343501] Lustre: server umount lustre-MDT0000 complete [ 7794.154126] LustreError: 255148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788672139 with bad export cookie 14065092713254967226 [ 7794.160457] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7794.166316] LustreError: 255148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7794.447049] Lustre: server umount lustre-MDT0001 complete [ 7806.588544] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 7809.593763] Key type lgssc unregistered [ 7809.944379] LNet: 260580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7809.958180] LNetError: 260580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7809.992941] LNet: Removed LNI 192.168.202.125@tcp [ 7811.052571] Key type .llcrypt unregistered [ 7811.054935] Key type ._llcrypt unregistered [ 7836.509483] Key type ._llcrypt registered [ 7836.510673] Key type .llcrypt registered [ 7836.636719] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 7838.059399] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7838.407689] alg: No test for adler32 (adler32-zlib) [ 7839.577958] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 7839.899855] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 7841.797150] Key type lgssc registered [ 7843.286974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7852.139571] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7852.172397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7853.721754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7858.137216] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 7863.274234] LustreError: 261997:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7863.291610] LustreError: 261997:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7866.868576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7867.356291] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7872.351851] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 7881.087152] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7888.006292] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7895.371798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7895.754489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7898.858865] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7902.131710] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 7909.516678] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7940.095148] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7940.104503] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7940.123336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7943.137172] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7943.144756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7943.154051] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7943.175134] Lustre: Skipped 1 previous similar message [ 7945.041594] Lustre: server umount lustre-OST0000 complete [ 7953.380798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7953.404543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7956.033760] Lustre: server umount lustre-MDT0000 complete [ 7958.497309] LustreError: 261992:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7958.513371] LustreError: 261992:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7959.709444] LustreError: 261978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788672304 with bad export cookie 2695732163495756410 [ 7959.715752] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7959.730101] LustreError: 261978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7960.266115] Lustre: server umount lustre-MDT0001 complete [ 7972.563112] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 7975.656530] Key type lgssc unregistered [ 7975.961311] LNet: 266307:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7975.967929] LNetError: 266307:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7975.983481] LNet: Removed LNI 192.168.202.125@tcp [ 7976.572490] Key type .llcrypt unregistered [ 7976.574259] Key type ._llcrypt unregistered [ 7996.504602] Key type ._llcrypt registered [ 7996.518271] Key type .llcrypt registered [ 7996.865634] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 7998.454451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7998.566704] alg: No test for adler32 (adler32-zlib) [ 7999.663886] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 7999.891779] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 8001.551686] Key type lgssc registered [ 8003.358077] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8012.208815] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8012.226649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8013.606225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8016.944645] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 8023.972268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8024.034479] LustreError: 267525:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8024.047492] LustreError: 267525:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8024.392683] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8029.265936] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 8037.027653] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8042.261919] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8050.031818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8050.760140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8058.223531] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 8058.941907] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8066.801201] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8084.452486] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8084.454050] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8084.463003] LustreError: Skipped 1 previous similar message [ 8084.464938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8084.493937] Lustre: Skipped 1 previous similar message [ 8086.060868] Lustre: server umount lustre-OST0000 complete [ 8089.572558] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8089.583884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8089.590944] Lustre: Skipped 1 previous similar message [ 8089.608209] Lustre: Skipped 2 previous similar messages [ 8094.691199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8094.695370] Lustre: Skipped 1 previous similar message [ 8095.992613] Lustre: server umount lustre-MDT0000 complete [ 8099.631638] LustreError: 267506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788672444 with bad export cookie 11626699090988068499 [ 8099.633664] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8099.648364] LustreError: 267506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8100.027353] Lustre: server umount lustre-MDT0001 complete [ 8112.734826] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 8115.781278] Key type lgssc unregistered [ 8116.127716] LNet: 271200:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8116.147308] LNetError: 271200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8116.162902] LNet: Removed LNI 192.168.202.125@tcp [ 8117.290541] Key type .llcrypt unregistered [ 8117.292838] Key type ._llcrypt unregistered [ 8146.227697] Key type ._llcrypt registered [ 8146.230215] Key type .llcrypt registered [ 8146.329661] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 8147.406368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8147.448110] alg: No test for adler32 (adler32-zlib) [ 8148.413048] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 8148.652956] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 8150.328580] Key type lgssc registered [ 8151.335926] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8160.218279] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8160.232158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8161.795120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8166.595468] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 8172.005685] LustreError: 272618:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8172.018864] LustreError: 272618:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8175.084934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8175.562493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8181.013775] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 8189.790658] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8196.564444] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8203.949228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8204.312598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8206.404718] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8211.099203] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 8218.716554] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8246.252627] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8246.281484] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8246.283503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8246.319176] Lustre: Skipped 1 previous similar message [ 8249.038320] Lustre: server umount lustre-OST0000 complete [ 8256.484668] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8256.493173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8256.510184] Lustre: Skipped 2 previous similar messages [ 8256.523100] Lustre: Skipped 1 previous similar message [ 8261.600847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8261.617310] Lustre: Skipped 1 previous similar message [ 8266.722406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8266.734775] Lustre: Skipped 1 previous similar message [ 8267.231586] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8267.486249] Lustre: server umount lustre-MDT0000 complete [ 8271.227060] LustreError: 273351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788672616 with bad export cookie 17164199481927709167 [ 8271.233221] LustreError: 273351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8271.233334] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8271.625578] Lustre: server umount lustre-MDT0001 complete [ 8282.810467] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 8286.125842] Key type lgssc unregistered [ 8286.497613] LNet: 276923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8286.511610] LNetError: 276923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8287.590041] LNet: Removed LNI 192.168.202.125@tcp [ 8288.502960] Key type .llcrypt unregistered [ 8288.505426] Key type ._llcrypt unregistered [ 8307.186950] Key type ._llcrypt registered [ 8307.189399] Key type .llcrypt registered [ 8307.357623] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 8308.736459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8308.862799] alg: No test for adler32 (adler32-zlib) [ 8310.045544] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 8310.288491] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 8312.023167] Key type lgssc registered [ 8313.617331] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8322.621846] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8322.646418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8324.286100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8330.034584] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 8334.819627] LustreError: 278159:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8334.834494] LustreError: 278159:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8339.462672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8339.893595] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8344.680681] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 8352.643115] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8358.574632] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8366.920810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8367.163439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8369.203221] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8375.316331] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 8388.421554] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8404.962703] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8404.962951] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8404.986976] LustreError: Skipped 1 previous similar message [ 8404.993667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8405.006029] Lustre: Skipped 1 previous similar message [ 8405.042682] Lustre: Skipped 1 previous similar message [ 8408.040968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8408.048655] Lustre: Skipped 1 previous similar message [ 8409.689874] Lustre: server umount lustre-OST0000 complete [ 8416.736627] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8416.747714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8416.754655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8418.284406] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8420.734724] Lustre: server umount lustre-MDT0000 complete [ 8423.399435] LustreError: 278151:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8423.444976] LustreError: 278151:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8424.984680] LustreError: 278124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788672769 with bad export cookie 3257659595938103038 [ 8424.997763] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8425.559920] Lustre: server umount lustre-MDT0001 complete [ 8439.305951] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 8442.524077] Key type lgssc unregistered [ 8442.958367] LNet: 281861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8442.974890] LNetError: 281861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8443.033046] LNet: Removed LNI 192.168.202.125@tcp [ 8444.075754] Key type .llcrypt unregistered [ 8444.078185] Key type ._llcrypt unregistered [ 8457.314115] Key type ._llcrypt registered [ 8457.318664] Key type .llcrypt registered [ 8457.513607] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8477.658549] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_hostid [ 8494.515791] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 8496.221184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8496.257569] alg: No test for adler32 (adler32-zlib) [ 8497.478845] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 8497.799627] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 8499.479695] Key type lgssc registered [ 8500.907870] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8541.965805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8542.027824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8543.369527] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8543.396202] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8543.506925] Lustre: lustre-MDT0000: new disk, initializing [ 8543.659669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8543.690706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8548.951655] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 8560.644032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8560.775475] Lustre: 285411:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8560.870403] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8560.873387] Lustre: Skipped 1 previous similar message [ 8560.990701] Lustre: lustre-MDT0001: new disk, initializing [ 8561.104349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8561.144708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8561.159662] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8565.865850] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 8576.928271] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8583.615861] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8590.726033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8591.032965] Lustre: lustre-OST0000: new disk, initializing [ 8591.036530] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8591.046123] Lustre: 287382:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8591.204559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8598.612939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8598.626105] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8598.767288] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8598.877427] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 8608.961549] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8614.495312] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8614.698464] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8619.593541] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8619.868116] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8624.613368] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8624.617887] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8624.652310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8629.731080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8629.737619] Lustre: Skipped 2 previous similar messages [ 8634.850706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8634.860730] Lustre: Skipped 1 previous similar message [ 8636.386415] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8636.602565] Lustre: server umount lustre-OST0000 complete [ 8643.040910] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8643.056673] LustreError: Skipped 1 previous similar message [ 8643.068437] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8643.096490] Lustre: Skipped 1 previous similar message [ 8643.111498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8645.091731] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8647.864382] Lustre: server umount lustre-MDT0000 complete [ 8651.992917] LustreError: 285403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788672996 with bad export cookie 16389254519082786194 [ 8651.995594] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8652.014242] LustreError: 285403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8652.472917] Lustre: server umount lustre-MDT0001 complete [ 8663.220835] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 01:36:47 (1788673007) [ 8670.470859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8670.944070] LustreError: 289796:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8670.957449] LustreError: 289796:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8670.994618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8675.684836] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 8676.324838] LustreError: 289797:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8680.416919] LustreError: 289796:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8684.733221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8685.221040] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8690.884355] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 8700.226877] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8706.855883] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8715.243125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8715.770665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8723.839262] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 8732.415219] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8755.686047] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8755.694814] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8755.704474] LustreError: Skipped 1 previous similar message [ 8755.705235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8755.705240] Lustre: Skipped 2 previous similar messages [ 8755.758679] Lustre: Skipped 1 previous similar message [ 8757.475284] Lustre: server umount lustre-OST0000 complete [ 8761.840224] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8761.853451] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8764.109256] Lustre: server umount lustre-MDT0000 complete [ 8765.928295] LustreError: 289797:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8765.955956] LustreError: 289797:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8768.901890] LustreError: 289779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788673113 with bad export cookie 16389254519082787069 [ 8768.915036] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8768.917503] LustreError: 289779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8769.389059] Lustre: server umount lustre-MDT0001 complete [ 8785.179977] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 8789.414322] Key type lgssc unregistered [ 8789.946787] LNet: 293354:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8789.955288] LNetError: 293354:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8789.973058] LNet: Removed LNI 192.168.202.125@tcp [ 8791.074060] Key type .llcrypt unregistered [ 8791.075612] Key type ._llcrypt unregistered [ 8823.243440] Key type ._llcrypt registered [ 8823.250208] Key type .llcrypt registered [ 8823.433069] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_hostid [ 8840.809845] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 8842.011465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8842.042447] alg: No test for adler32 (adler32-zlib) [ 8843.552871] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 8843.895385] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 8845.639337] Key type lgssc registered [ 8847.121633] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8904.280497] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8904.308728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8905.541635] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8905.592050] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8905.678117] Lustre: lustre-MDT0000: new disk, initializing [ 8905.785421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8905.823536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8911.494052] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 8923.482362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8923.643081] Lustre: 297178:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8923.702969] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8923.712230] Lustre: Skipped 1 previous similar message [ 8923.795212] Lustre: lustre-MDT0001: new disk, initializing [ 8923.950111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8924.038409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8924.058175] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8929.287660] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 8938.907744] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8945.358510] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8954.616810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8954.864369] Lustre: lustre-OST2710: new disk, initializing [ 8954.867755] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8954.873854] Lustre: 299148:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8954.964621] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8961.661676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8961.689407] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8961.871420] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8963.420046] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 8974.373478] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8982.626346] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8982.820494] Lustre: lustre-OST03e8: new disk, initializing [ 8982.823136] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8982.826298] Lustre: 300213:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8982.899242] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8990.449972] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 8991.909692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8991.930170] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8992.029659] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8999.055348] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9008.245315] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9008.580994] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9013.829849] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9014.067887] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9020.082710] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9020.362919] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9025.744860] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9026.044614] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9032.200641] Lustre: server umount lustre-MDT0000 complete [ 9033.193280] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9033.220414] Lustre: Skipped 1 previous similar message [ 9036.432462] LustreError: 297170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788673381 with bad export cookie 13030641819217328936 [ 9036.438571] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9036.451028] LustreError: 297170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9036.915379] Lustre: server umount lustre-MDT0001 complete [ 9051.796273] Lustre: server umount lustre-OST2710 complete [ 9054.687131] Lustre: 294975:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788673383/real 1788673383] req@00000000cdeeeefc x1875559776852608/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788673399 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9054.722196] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9054.738383] Lustre: Skipped 2 previous similar messages [ 9056.640804] Lustre: server umount lustre-OST03e8 complete [ 9080.806898] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_hostid [ 9098.336544] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 9142.009692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9142.401335] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9142.467960] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9142.569420] Lustre: lustre-MDT0000: new disk, initializing [ 9142.736603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9142.769093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9148.586202] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 9160.935638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9161.207858] Lustre: 305491:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9161.313870] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9161.322883] Lustre: Skipped 1 previous similar message [ 9161.458253] Lustre: lustre-MDT0001: new disk, initializing [ 9161.564180] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9161.602884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9161.611456] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9166.319232] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 9176.962414] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9184.051678] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9191.404192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9191.764373] Lustre: lustre-OST0000: new disk, initializing [ 9191.770186] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9191.773745] Lustre: 307460:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9191.891362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9199.928543] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 9200.721041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9200.734639] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9200.868685] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9211.289857] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9217.417682] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9217.705486] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9223.194472] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9223.517199] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9226.721254] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9226.733337] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9226.746986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9226.755588] Lustre: Skipped 1 previous similar message [ 9226.767631] Lustre: Skipped 1 previous similar message [ 9231.546523] Lustre: server umount lustre-OST0000 complete [ 9236.962456] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9236.975912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9241.977216] Lustre: server umount lustre-MDT0000 complete [ 9242.080557] LustreError: 305496:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9242.104945] LustreError: 305496:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9245.708250] LustreError: 307459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788673590 with bad export cookie 13030641819217330609 [ 9245.713741] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9245.723507] LustreError: 307459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9246.076496] Lustre: server umount lustre-MDT0001 complete [ 9256.176746] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 01:46:40 (1788673600) [ 9258.027820] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9259.626135] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 01:46:44 (1788673604) [ 9263.464976] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 9293.708468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9293.909765] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9293.925507] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9294.060742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9299.585983] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 9309.515486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9309.625337] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9309.664601] Lustre: 311314:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9315.885210] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 9341.052769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9341.508463] Lustre: Denying initial registration attempt from nid 192.168.202.125@tcp, specified as failover [ 9341.523780] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9341.537033] LustreError: 312772:0:(tgt_mount.c:2619:server_fill_super()) Unable to start targets: -99 [ 9341.552832] LustreError: 312772:0:(tgt_mount.c:2134:server_put_super()) no obd lustre-OST0000 [ 9341.558480] LustreError: 312772:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9341.649818] Lustre: server umount lustre-OST0000 complete [ 9341.654892] LustreError: 312772:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9351.136077] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9351.151841] LustreError: Skipped 1 previous similar message [ 9351.153869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9351.169789] Lustre: Skipped 1 previous similar message [ 9351.173516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9351.175578] Lustre: Skipped 1 previous similar message [ 9353.703675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9353.712970] Lustre: Skipped 1 previous similar message [ 9355.459807] Lustre: server umount lustre-MDT0000 complete [ 9358.823612] LustreError: 312063:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9358.839659] LustreError: 312063:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9359.382905] LustreError: 311308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788673704 with bad export cookie 13030641819217331498 [ 9359.386809] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9359.400339] LustreError: 311308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9360.058392] Lustre: server umount lustre-MDT0001 complete [ 9373.145134] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 9377.260888] Key type lgssc unregistered [ 9377.783723] LNet: 313893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9377.792668] LNetError: 313893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9377.817415] LNet: Removed LNI 192.168.202.125@tcp [ 9379.247772] Key type .llcrypt unregistered [ 9379.249625] Key type ._llcrypt unregistered [ 9394.159736] Key type ._llcrypt registered [ 9394.162809] Key type .llcrypt registered [ 9394.277924] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 9395.757068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9395.945225] alg: No test for adler32 (adler32-zlib) [ 9397.174963] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 9397.509820] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 9399.207359] Key type lgssc registered [ 9400.742980] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9451.840457] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 9461.287783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9461.338147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9462.826741] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9462.868842] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9463.138430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9467.699185] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 9477.429255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9477.524737] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9477.554287] Lustre: 316660:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9477.949398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9482.841532] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 9491.070783] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9496.987360] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9517.049571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9517.399698] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9517.561733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9524.583027] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 9534.745905] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9537.506931] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9537.508819] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9537.529621] LustreError: Skipped 1 previous similar message [ 9537.561726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9542.627586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9542.636612] Lustre: Skipped 1 previous similar message [ 9543.611479] Lustre: server umount lustre-OST0000 complete [ 9547.744801] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9547.754661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9547.760654] Lustre: Skipped 2 previous similar messages [ 9547.774705] Lustre: Skipped 2 previous similar messages [ 9552.871281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9552.889427] Lustre: Skipped 1 previous similar message [ 9554.055977] Lustre: server umount lustre-MDT0000 complete [ 9558.359722] LustreError: 318524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788673903 with bad export cookie 1900896901380063245 [ 9558.377699] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9558.380262] LustreError: 318524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9558.918338] Lustre: server umount lustre-MDT0001 complete [ 9570.978727] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 9574.282295] Key type lgssc unregistered [ 9574.561166] LNet: 320232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9574.574334] LNetError: 320232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9575.656554] LNet: Removed LNI 192.168.202.125@tcp [ 9576.777539] Key type .llcrypt unregistered [ 9576.780131] Key type ._llcrypt unregistered [ 9603.558948] Key type ._llcrypt registered [ 9603.561135] Key type .llcrypt registered [ 9603.670120] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 9604.657752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9604.701686] alg: No test for adler32 (adler32-zlib) [ 9606.098120] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 9606.448898] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 9608.255308] Key type lgssc registered [ 9609.513248] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9617.315418] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9617.352722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9618.993088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9625.401485] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 9628.134576] LustreError: 321646:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9628.165189] LustreError: 321646:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9633.255122] LustreError: 321647:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9635.763571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9636.135583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9641.309591] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 9648.692660] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9655.559448] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9665.014216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9665.637315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9672.664757] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 9680.028799] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9698.787422] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9698.792969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9698.800990] Lustre: Skipped 2 previous similar messages [ 9698.817405] Lustre: Skipped 2 previous similar messages [ 9701.154139] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.25@tcp (stopping) [ 9703.904918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9704.338505] Lustre: server umount lustre-MDT0000 complete [ 9706.241892] LustreError: 322743:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9708.831755] LustreError: 321627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788674053 with bad export cookie 9300120792002382619 [ 9708.854864] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9708.856524] LustreError: 321627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9709.025211] LustreError: 321642:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9709.040517] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9709.079702] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9709.089306] Lustre: Skipped 2 previous similar messages [ 9713.061096] LustreError: 322743:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9713.089407] LustreError: 322743:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9716.489644] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.25@tcp (stopping) [ 9716.496639] Lustre: Skipped 2 previous similar messages [ 9721.595462] LustreError: 321642:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9721.614766] LustreError: 321642:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9723.359864] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9723.615644] Lustre: server umount lustre-MDT0001 complete [ 9729.456983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9739.838246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9753.057629] LustreError: 320954:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000885c3f13 x1875560576886656/t0(0) o250->MGC192.168.202.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9753.588795] LustreError: 323479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9753.606889] LustreError: 323479:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9753.670271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9753.736208] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9758.463792] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 9768.957574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9769.263220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9775.055153] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 9778.684666] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9782.058255] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9783.659842] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9789.298760] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9791.193581] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9799.141468] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9799.147533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9799.151117] Lustre: Skipped 1 previous similar message [ 9799.170574] Lustre: Skipped 5 previous similar messages [ 9802.968436] Lustre: server umount lustre-OST0000 complete [ 9809.379570] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9809.403777] Lustre: Skipped 1 previous similar message [ 9819.620812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9819.626379] Lustre: Skipped 5 previous similar messages [ 9822.175285] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9822.409362] Lustre: server umount lustre-MDT0000 complete [ 9824.749377] LustreError: 325274:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9824.763209] LustreError: 325274:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9826.349305] LustreError: 324458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788674171 with bad export cookie 9300120792002390151 [ 9826.360467] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9826.367529] LustreError: 324458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9826.780509] Lustre: server umount lustre-MDT0001 complete [ 9841.028504] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 9845.500682] Key type lgssc unregistered [ 9845.963659] LNet: 327945:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9845.979441] LNetError: 327945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9846.008060] LNet: Removed LNI 192.168.202.125@tcp [ 9847.286328] Key type .llcrypt unregistered [ 9847.300585] Key type ._llcrypt unregistered [ 9877.487673] Key type ._llcrypt registered [ 9877.489974] Key type .llcrypt registered [ 9877.583434] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 9878.734870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9878.945867] alg: No test for adler32 (adler32-zlib) [ 9880.187391] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 9880.569772] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 9882.319668] Key type lgssc registered [ 9883.983855] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9895.619922] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9895.646405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9897.300610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9903.611054] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 9907.684823] LustreError: 329364:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9907.708150] LustreError: 329364:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9912.805270] LustreError: 329365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9914.160547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9914.830379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9920.423878] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 9929.767354] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9936.898221] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9944.546254] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9944.575950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9944.580650] Lustre: Skipped 1 previous similar message [ 9947.120853] Lustre: server umount lustre-MDT0000 complete [ 9949.664825] LustreError: 329359:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9949.692260] LustreError: 329359:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9951.326885] LustreError: 329345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788674296 with bad export cookie 2483388286808489491 [ 9951.342291] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9951.351315] LustreError: 329345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9951.802845] Lustre: server umount lustre-MDT0001 complete [ 9968.029390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9968.324863] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9968.384759] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9968.611970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9973.095184] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 9982.440609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9982.563231] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9982.592671] Lustre: 331951:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9983.008185] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9988.042867] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 9996.172799] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10002.670830] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10013.350336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10013.521392] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10013.530965] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10013.539741] LustreError: 333837:0:(tgt_mount.c:2619:server_fill_super()) Unable to start targets: -2 [10013.544736] LustreError: 333837:0:(tgt_mount.c:2134:server_put_super()) no obd lustre-OST0000 [10013.549248] LustreError: 333837:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10013.622769] Lustre: server umount lustre-OST0000 complete [10013.626305] LustreError: 333837:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10024.362532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10024.619465] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10024.891684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10030.910372] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [10034.365819] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10038.606599] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10047.689861] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10047.806580] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10047.926470] Lustre: lustre-OST0001: new disk, initializing [10047.933690] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10047.943227] Lustre: 335378:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10048.109697] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10056.412343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10056.436784] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10056.528269] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10057.063863] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [10066.489235] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10072.039659] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10072.049071] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10072.073142] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10072.083776] Lustre: Skipped 2 previous similar messages [10072.111042] Lustre: Skipped 1 previous similar message [10077.160385] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10077.165719] Lustre: Skipped 1 previous similar message [10082.289873] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10082.315689] Lustre: Skipped 1 previous similar message [10085.343361] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10085.471727] Lustre: server umount lustre-OST0001 complete [10087.415297] LustreError: 334305:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10087.437744] LustreError: 334305:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10090.465486] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10090.472062] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10090.488267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10092.522279] LustreError: 335706:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10092.552545] LustreError: 335706:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10102.766601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10102.767106] LustreError: 334306:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10102.795020] Lustre: Skipped 6 previous similar messages [10102.842055] LustreError: 334306:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10104.287120] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10104.572222] Lustre: server umount lustre-OST0000 complete [10110.945479] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10110.952928] LustreError: Skipped 1 previous similar message [10110.956521] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10110.970591] Lustre: Skipped 1 previous similar message [10115.644397] Lustre: server umount lustre-MDT0000 complete [10120.482828] LustreError: 331944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788674465 with bad export cookie 2483388286808490044 [10120.489533] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10120.497223] LustreError: 331944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10120.941176] Lustre: server umount lustre-MDT0001 complete [10134.055146] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [10137.798994] Key type lgssc unregistered [10138.253328] LNet: 337349:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10138.257803] LNetError: 337349:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10138.284454] LNet: Removed LNI 192.168.202.125@tcp [10139.555869] Key type .llcrypt unregistered [10139.557229] Key type ._llcrypt unregistered [10218.809286] Key type ._llcrypt registered [10218.818815] Key type .llcrypt registered [10218.972056] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_hostid [10237.092911] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [10238.876826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10238.923334] alg: No test for adler32 (adler32-zlib) [10240.081327] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [10240.427091] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [10242.183214] Key type lgssc registered [10244.178218] Lustre: Echo OBD driver; http://www.lustre.org/ [10283.404438] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10283.454652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10284.841664] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10284.903852] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10285.061665] Lustre: lustre-MDT0000: new disk, initializing [10285.197610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10285.244433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10290.182491] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [10300.430281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10300.780149] Lustre: 342283:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10300.947298] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10300.961242] Lustre: Skipped 1 previous similar message [10301.077855] Lustre: lustre-MDT0001: new disk, initializing [10301.270391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10301.352202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10301.380977] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10307.650404] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [10317.811309] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10324.175932] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10331.734552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10331.989215] Lustre: lustre-OST0000: new disk, initializing [10331.991980] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10331.997111] Lustre: 344252:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10332.113759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10338.701835] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [10341.533298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10341.548620] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10341.650926] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10348.576786] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10354.640764] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10354.891643] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10361.137865] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10361.484969] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10367.457089] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10367.458463] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10367.471774] LustreError: Skipped 1 previous similar message [10367.494829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10367.508609] Lustre: Skipped 1 previous similar message [10372.577770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10372.603593] Lustre: Skipped 1 previous similar message [10377.698727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10378.208821] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10378.422905] Lustre: server umount lustre-OST0000 complete [10382.816559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10382.821383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10382.829393] Lustre: Skipped 2 previous similar messages [10382.840367] Lustre: Skipped 2 previous similar messages [10387.936681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10387.949919] Lustre: Skipped 1 previous similar message [10397.155498] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10397.791280] Lustre: server umount lustre-MDT0000 complete [10402.114912] LustreError: 342273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788674747 with bad export cookie 14178982816114333120 [10402.129956] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10402.131492] LustreError: 342273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10402.694922] Lustre: server umount lustre-MDT0001 complete [10412.987623] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 02:05:57 (1788674757) [10420.108118] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 02:06:04 (1788674764) [10430.039946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10430.580628] LustreError: 347010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10430.615199] LustreError: 347010:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10430.728141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10436.067964] LustreError: 347011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10436.712861] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [10440.162932] LustreError: 347010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10445.281562] LustreError: 347011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10446.602889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10446.957154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10452.425972] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [10462.293928] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10469.705483] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10477.906478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10478.340968] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10485.182944] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [10494.480391] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10508.057466] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10510.053039] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10511.811452] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10513.678165] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10515.376294] Lustre: Failing over lustre-MDT0000 [10516.452594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10516.458902] LustreError: 348851:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10516.473463] Lustre: Skipped 2 previous similar messages [10516.494568] LustreError: 348851:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10517.673575] Lustre: server umount lustre-MDT0000 complete [10526.705512] LustreError: 347011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10526.737464] LustreError: 347011:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10532.842605] Lustre: 340506:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788674861/real 1788674861] req@00000000c57d14b4 x1875561241718016/t0(0) o400->MGC192.168.202.125@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788674877 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10532.872448] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10536.403984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10542.063571] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xc4c5ddbf6edcdd45 [10542.073961] Lustre: MGC192.168.202.125@tcp: Connection restored to 0@lo (at 0@lo) [10542.340308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10542.388344] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10546.989719] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [10547.452623] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10547.683109] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10547.707508] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10547.754594] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10555.867044] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10557.651816] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10562.130350] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10564.792113] Lustre: Failing over lustre-MDT0000 [10565.310580] Lustre: server umount lustre-MDT0000 complete [10567.964657] LustreError: 347006:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10567.981727] LustreError: 347006:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [10568.160834] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10568.164231] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10568.205088] Lustre: Skipped 2 previous similar messages [10584.614220] Lustre: 340507:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788674913/real 1788674913] req@00000000408b9d52 x1875561241739136/t0(0) o400->MGC192.168.202.125@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788674929 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10584.668573] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10586.544663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10594.277684] LustreError: 340503:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b5c9dbd8 x1875561241744512/t0(0) o250->MGC192.168.202.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10594.807772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10594.898085] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10599.166071] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10599.925658] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10599.935569] Lustre: Skipped 2 previous similar messages [10599.978152] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10600.028592] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10600.264259] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [10610.313550] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10612.003468] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10620.386756] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10620.403844] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10620.408245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10620.408254] Lustre: Skipped 2 previous similar messages [10620.435235] Lustre: Skipped 2 previous similar messages [10624.137272] Lustre: server umount lustre-MDT0000 complete [10625.513968] LustreError: 347011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10625.530791] LustreError: 347011:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 29 previous similar messages [10627.325698] LustreError: 346992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788674972 with bad export cookie 14178982816114335836 [10627.325698] LustreError: 346990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788674972 with bad export cookie 14178982816114335836 [10627.326574] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10627.339504] LustreError: 346992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10627.670555] Lustre: server umount lustre-MDT0001 complete [10641.663712] Lustre: server umount lustre-OST0000 complete [10658.563134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10659.454051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10666.210980] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [10675.900435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10681.651782] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [10689.869765] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10695.697637] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10702.462062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10704.844856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10708.755284] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [10716.624815] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10726.371215] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10726.387570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10726.393202] Lustre: Skipped 3 previous similar messages [10728.362969] Lustre: server umount lustre-OST0000 complete [10733.025565] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10738.865499] Lustre: server umount lustre-MDT0000 complete [10741.731925] LustreError: 353376:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10741.763397] LustreError: 353376:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10743.200940] LustreError: 353358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788675088 with bad export cookie 14178982816114336578 [10743.214611] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10743.771826] Lustre: server umount lustre-MDT0001 complete [10755.120696] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [10758.402365] Key type lgssc unregistered [10758.740630] LNet: 356907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10758.746675] LNetError: 356907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10758.757358] LNet: Removed LNI 192.168.202.125@tcp [10759.865126] Key type .llcrypt unregistered [10759.866746] Key type ._llcrypt unregistered [10789.310553] Key type ._llcrypt registered [10789.311885] Key type .llcrypt registered [10789.422702] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [10790.978180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10791.003186] alg: No test for adler32 (adler32-zlib) [10792.048848] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [10792.301264] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [10794.095520] Key type lgssc registered [10795.166820] Lustre: Echo OBD driver; http://www.lustre.org/ [10802.720422] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10802.736331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10804.312763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10811.114951] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [10814.438488] LustreError: 358419:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10814.464487] LustreError: 358419:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10819.555908] LustreError: 358418:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10820.766227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10821.275331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10826.125606] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [10833.613760] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10839.843199] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10846.406964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10846.774237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10849.846369] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10852.920792] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [10860.322668] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10877.408953] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10877.420455] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10877.437844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10881.507975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10881.526715] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10881.545948] Lustre: Skipped 1 previous similar message [10883.033672] Lustre: server umount lustre-MDT0000 complete [10886.255989] LustreError: 358400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788675231 with bad export cookie 3694321312654205508 [10886.274608] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10886.626027] LustreError: 358415:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10886.629315] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10886.835905] Lustre: server umount lustre-MDT0001 complete [10892.742568] Lustre: server umount lustre-OST0000 complete [10906.502416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10906.847111] LustreError: 362159:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10906.863100] LustreError: 362159:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10906.919537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10911.699549] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [10912.226243] LustreError: 362160:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10921.369327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10921.440573] LustreError: 362159:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10921.458162] LustreError: 362159:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10921.914126] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10927.240714] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [10936.833243] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10943.238436] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10951.166381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10951.650566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10952.691062] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10959.305398] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [10967.464834] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10982.372776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10982.373485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10982.386840] Lustre: Skipped 2 previous similar messages [10982.397077] Lustre: Skipped 4 previous similar messages [10984.808091] Lustre: server umount lustre-MDT0000 complete [10987.492231] LustreError: 362160:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10987.510933] LustreError: 362160:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10989.746725] LustreError: 362141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788675334 with bad export cookie 3694321312654206614 [10989.769867] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10989.770958] LustreError: 362141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [10990.173129] Lustre: server umount lustre-MDT0001 complete [11004.890349] Lustre: server umount lustre-OST0000 complete [11024.395685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11025.102502] LustreError: 366071:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11025.230828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11031.401138] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [11043.901523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11049.708124] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [11058.870323] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11065.584222] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11073.678829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11073.992134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11073.998687] Lustre: Skipped 1 previous similar message [11081.197938] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [11090.207303] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11096.558653] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11106.799768] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11106.801563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11106.802162] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11106.821273] Lustre: Skipped 1 previous similar message [11106.835162] Lustre: Skipped 1 previous similar message [11111.901615] Lustre: server umount lustre-OST0000 complete [11116.003226] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11116.011442] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11116.024296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11126.241751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11126.248680] Lustre: Skipped 5 previous similar messages [11130.335274] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11130.497700] Lustre: server umount lustre-MDT0000 complete [11131.366088] LustreError: 366067:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11131.393929] LustreError: 366067:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11133.916126] LustreError: 366051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788675478 with bad export cookie 3694321312654207643 [11133.928937] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11133.937289] LustreError: 366051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11134.284073] Lustre: server umount lustre-MDT0001 complete [11144.809511] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [11147.942833] Key type lgssc unregistered [11148.238205] LNet: 369624:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11148.251860] LNetError: 369624:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11148.279810] LNet: Removed LNI 192.168.202.125@tcp [11149.396621] Key type .llcrypt unregistered [11149.401691] Key type ._llcrypt unregistered [11178.785718] Key type ._llcrypt registered [11178.790093] Key type .llcrypt registered [11178.960275] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [11180.718445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11180.754752] alg: No test for adler32 (adler32-zlib) [11181.917856] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [11182.143442] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [11183.799208] Key type lgssc registered [11185.550421] Lustre: Echo OBD driver; http://www.lustre.org/ [11194.826453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11194.840086] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11194.845288] LustreError: 371061:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11194.858718] LustreError: 371061:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11194.863900] LustreError: 371061:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11194.869487] LustreError: 371061:0:(tgt_mount.c:2587:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11194.875517] LustreError: 371061:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11202.738523] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11202.744736] LustreError: 371460:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11202.767950] LustreError: 371460:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11202.777894] LustreError: 371460:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11202.784593] LustreError: 371460:0:(tgt_mount.c:2587:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11202.797495] LustreError: 371460:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11225.219720] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [11228.706284] Key type lgssc unregistered [11229.060319] LNet: 372463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11229.072456] LNetError: 372463:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11229.088472] LNet: Removed LNI 192.168.202.125@tcp [11230.068740] Key type .llcrypt unregistered [11230.073558] Key type ._llcrypt unregistered [11248.704498] Key type ._llcrypt registered [11248.707302] Key type .llcrypt registered [11248.844924] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_hostid [11263.422390] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [11264.520808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11264.550500] alg: No test for adler32 (adler32-zlib) [11265.634192] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [11265.931729] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [11267.607206] Key type lgssc registered [11268.706275] Lustre: Echo OBD driver; http://www.lustre.org/ [11306.549123] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11306.565495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11307.830672] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11307.858958] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11307.972615] Lustre: lustre-MDT0000: new disk, initializing [11308.086351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11308.127960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11313.727402] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [11325.217050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11325.381492] Lustre: 375617:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11325.480151] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11325.483575] Lustre: Skipped 1 previous similar message [11325.595537] Lustre: lustre-MDT0001: new disk, initializing [11325.776783] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11325.838395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11325.848937] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11331.717729] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [11342.521060] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11348.928915] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11356.112167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11356.351608] Lustre: lustre-OST0000: new disk, initializing [11356.362789] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11356.367703] Lustre: 377586:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11356.474517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11363.437528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11363.445243] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [11363.447884] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11363.620669] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11373.930990] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11380.196448] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11380.397731] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11386.024766] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11386.376482] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11389.411810] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11389.413744] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11389.436996] LustreError: Skipped 1 previous similar message [11389.443388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11389.475179] Lustre: Skipped 1 previous similar message [11389.500672] Lustre: Skipped 1 previous similar message [11394.534491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11394.542359] Lustre: Skipped 1 previous similar message [11395.232914] Lustre: server umount lustre-OST0000 complete [11399.648746] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11399.660673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11399.671911] Lustre: Skipped 1 previous similar message [11404.768306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11404.782437] Lustre: Skipped 1 previous similar message [11405.919728] Lustre: server umount lustre-MDT0000 complete [11409.849634] LustreError: 375609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788675754 with bad export cookie 2414883746201839589 [11409.866017] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11409.873732] LustreError: 375609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11410.238660] Lustre: server umount lustre-MDT0001 complete [11419.265788] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11421.303580] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 02:22:45 (1788675765) [11428.810023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11429.604770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11434.244555] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [11434.984245] LustreError: 380045:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11434.998887] LustreError: 380045:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11439.074746] LustreError: 380046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11444.192904] LustreError: 380045:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11446.594348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11447.326233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11453.742283] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [11463.274875] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11469.227286] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11476.282033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11476.545465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11482.932132] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [11490.520343] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11498.353085] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11498.547131] Lustre: lustre-OST0001: new disk, initializing [11498.551622] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11498.556949] Lustre: 382884:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11498.694609] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11505.664228] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [11507.346496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11507.353588] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11507.471898] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11515.067369] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11521.837832] Lustre: lustre-OST0001: Not available for connect from 192.168.202.25@tcp (stopping) [11523.047563] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11523.072682] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11523.091649] Lustre: Skipped 2 previous similar messages [11525.284193] Lustre: server umount lustre-OST0001 complete [11526.902530] LustreError: 381868:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11533.283057] LustreError: 381868:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11533.307415] LustreError: 381868:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11543.536595] LustreError: 383194:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11543.543538] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11543.553249] LustreError: 383194:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11543.573770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11543.582776] Lustre: Skipped 3 previous similar messages [11548.767500] Lustre: server umount lustre-OST0000 complete [11553.761182] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11553.782811] Lustre: Skipped 2 previous similar messages [11559.755665] Lustre: server umount lustre-MDT0000 complete [11564.008644] LustreError: 380046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11564.028928] LustreError: 380046:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11564.041542] LustreError: 380026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788675908 with bad export cookie 2414883746201840471 [11564.046248] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11564.052235] LustreError: 380026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11578.336114] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11578.676142] Lustre: server umount lustre-MDT0001 complete [11591.234616] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [11594.359877] Key type lgssc unregistered [11594.764619] LNet: 384874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11594.778299] LNetError: 384874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11594.800385] LNet: Removed LNI 192.168.202.125@tcp [11595.667588] Key type .llcrypt unregistered [11595.670876] Key type ._llcrypt unregistered [11648.039332] Key type ._llcrypt registered [11648.040582] Key type .llcrypt registered [11648.132864] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [11649.222963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11649.275141] alg: No test for adler32 (adler32-zlib) [11650.413910] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [11650.625729] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [11652.277412] Key type lgssc registered [11653.572718] Lustre: Echo OBD driver; http://www.lustre.org/ [11661.650415] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11661.665937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11663.019658] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11663.045675] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11663.270317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11667.249905] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [11676.149161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11676.304217] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11676.352928] Lustre: 387672:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11676.683725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11681.472872] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [11689.603853] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11695.461905] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11702.245130] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11702.246484] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11702.266337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11702.289183] Lustre: Skipped 1 previous similar message [11704.484093] Lustre: server umount lustre-MDT0000 complete [11708.632877] LustreError: 388413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788676053 with bad export cookie 6363957288983946639 [11708.638252] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11708.654445] LustreError: 388413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11709.174903] Lustre: server umount lustre-MDT0001 complete [11720.022621] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 02:27:44 (1788676064) [11727.659551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11728.253583] LustreError: 390183:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11728.279570] LustreError: 390183:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11728.362700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11733.477367] LustreError: 390184:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11733.888581] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [11738.593458] LustreError: 390183:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11743.712310] LustreError: 390184:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11744.622429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11745.060736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11750.453994] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [11757.922640] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11763.474820] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11770.175922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11770.315737] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11770.473606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11776.046835] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [11783.351207] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11790.564361] Lustre: Permanently deactivating lustre-OST0000 [11790.591669] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11797.554992] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11799.974621] LustreError: 393122:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11799.997840] LustreError: 393122:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11800.004297] LustreError: 393122:0:(mgs_handler.c:1247:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11804.592232] LustreError: 393171:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11804.598849] LustreError: 393171:0:(mgs_handler.c:1247:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11809.225542] Lustre: server umount lustre-OST0000 complete [11813.998765] LustreError: 393422:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11814.007806] LustreError: 393422:0:(mgs_handler.c:1247:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11816.929336] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11816.932475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11816.944693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11816.967888] Lustre: Skipped 2 previous similar messages [11816.997696] Lustre: Skipped 1 previous similar message [11822.052312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11822.068037] Lustre: Skipped 1 previous similar message [11826.253973] Lustre: server umount lustre-MDT0000 complete [11827.194931] LustreError: 390918:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11827.231903] LustreError: 390918:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11830.175247] LustreError: 392018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788676175 with bad export cookie 6363957288983947227 [11830.176495] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11830.195565] LustreError: 392018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11840.653376] Lustre: server umount lustre-MDT0001 complete [11849.704918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11854.471555] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [11857.215653] LustreError: 394581:0:(mgs_handler.c:1247:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11863.538455] Lustre: 394920:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.125@tcp [11865.841459] Lustre: server umount MGS complete [11877.769525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11878.303351] LustreError: 395645:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11878.315668] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11878.324502] Lustre: Skipped 1 previous similar message [11878.362348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11883.448602] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [11893.960023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11894.334450] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11894.408190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11898.433268] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [11905.273552] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11913.021416] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11915.109309] Lustre: Permanently reactivating lustre-OST0000 [11915.143988] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11915.151855] Lustre: Skipped 2 previous similar messages [11944.931772] LustreError: 397441:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11953.547903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11954.010450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11961.388139] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [11970.076366] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11979.745703] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11979.752949] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11979.764101] LustreError: Skipped 1 previous similar message [11979.770665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11979.803385] Lustre: Skipped 1 previous similar message [11982.217661] Lustre: server umount lustre-OST0000 complete [11986.406236] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11986.422229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11986.455680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11986.471381] Lustre: Skipped 1 previous similar message [11992.408346] Lustre: server umount lustre-MDT0000 complete [11995.111552] LustreError: 395642:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11995.120128] LustreError: 395642:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11996.415057] LustreError: 395627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788676341 with bad export cookie 6363957288983948277 [11996.433857] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11996.802985] Lustre: server umount lustre-MDT0001 complete [12008.376625] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [12011.410607] Key type lgssc unregistered [12011.756274] LNet: 400009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12011.765800] LNetError: 400009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12011.794948] LNet: Removed LNI 192.168.202.125@tcp [12012.536180] Key type .llcrypt unregistered [12012.539270] Key type ._llcrypt unregistered [12030.323750] Key type ._llcrypt registered [12030.326528] Key type .llcrypt registered [12030.428189] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_hostid [12045.297963] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [12046.555905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12046.618491] alg: No test for adler32 (adler32-zlib) [12047.635386] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [12047.850234] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [12049.487203] Key type lgssc registered [12050.634315] Lustre: Echo OBD driver; http://www.lustre.org/ [12084.388162] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12084.424328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12085.640270] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12085.667127] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12085.758369] Lustre: lustre-MDT0000: new disk, initializing [12085.847189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12085.863942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12090.424474] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [12100.860307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12101.003566] Lustre: 403162:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12101.066794] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12101.070684] Lustre: Skipped 1 previous similar message [12101.144555] Lustre: lustre-MDT0001: new disk, initializing [12101.252660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12101.284101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12101.294632] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12106.670695] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [12117.067747] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12123.272461] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12130.148800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12130.505699] Lustre: lustre-OST0000: new disk, initializing [12130.509700] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12130.514718] Lustre: 405131:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12130.573798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12136.284280] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [12137.568854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12137.586884] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12137.733114] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12144.784723] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12149.347548] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12149.558606] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12154.449897] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12154.633121] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12158.441151] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12158.441927] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12158.447688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12158.462667] Lustre: Skipped 1 previous similar message [12162.796893] Lustre: server umount lustre-OST0000 complete [12168.165053] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12168.170455] LustreError: Skipped 1 previous similar message [12168.173368] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12168.182168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12168.185931] Lustre: Skipped 1 previous similar message [12172.418502] Lustre: server umount lustre-MDT0000 complete [12177.393990] LustreError: 405130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788676522 with bad export cookie 5695133203348091812 [12177.407078] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12177.878626] Lustre: server umount lustre-MDT0001 complete [12188.196814] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12190.327992] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 02:35:34 (1788676534) [12197.622734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12198.210171] LustreError: 407591:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12198.239075] LustreError: 407591:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12198.307485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12203.492626] LustreError: 407592:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12203.682329] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [12208.613037] LustreError: 407591:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12211.395738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12211.745168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12215.846874] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [12222.545271] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12228.359829] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12234.880504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12235.210556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12241.795590] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [12248.927280] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12250.648589] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12253.350067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12261.856466] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12261.865369] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12261.877765] Lustre: Skipped 1 previous similar message [12261.884157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12261.897222] Lustre: Skipped 1 previous similar message [12263.910355] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12263.917041] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12263.918223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12263.933476] Lustre: Skipped 1 previous similar message [12265.095376] Lustre: server umount lustre-OST0000 complete [12269.025332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12269.027641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12269.040759] Lustre: Skipped 1 previous similar message [12269.058814] Lustre: Skipped 1 previous similar message [12274.879467] Lustre: server umount lustre-MDT0000 complete [12278.617471] LustreError: 407573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788676623 with bad export cookie 5695133203348092694 [12278.628336] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12278.631935] LustreError: 407573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [12278.841695] Lustre: server umount lustre-MDT0001 complete [12290.327944] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [12293.855468] Key type lgssc unregistered [12294.236779] LNet: 411319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12294.247551] LNetError: 411319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12294.264678] LNet: Removed LNI 192.168.202.125@tcp [12295.220519] Key type .llcrypt unregistered [12295.224621] Key type ._llcrypt unregistered [12319.382965] Key type ._llcrypt registered [12319.384837] Key type .llcrypt registered [12319.540847] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [12319.813755] Key type .llcrypt unregistered [12319.815560] Key type ._llcrypt unregistered [12336.015495] Key type ._llcrypt registered [12336.016598] Key type .llcrypt registered [12336.080320] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [12337.085658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12337.114212] alg: No test for adler32 (adler32-zlib) [12338.275892] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [12338.498621] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [12340.191138] Key type lgssc registered [12341.086582] Lustre: Echo OBD driver; http://www.lustre.org/ [12347.252736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12347.268548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12348.568703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12351.881671] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [12358.052646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12358.114869] LustreError: 413523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12358.127706] LustreError: 413523:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12358.314905] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12363.555459] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [12363.765853] LustreError: 413524:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12363.795454] LustreError: 413524:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12363.796184] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12368.866848] LustreError: 413523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12368.884030] LustreError: 413523:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12369.946189] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12373.989449] LustreError: 413519:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12374.005704] LustreError: 413519:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12375.726764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12376.088161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12380.238919] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [12389.347817] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12389.353520] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12389.364070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12389.561610] Lustre: server umount lustre-OST0000 complete [12396.004704] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12396.017515] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12396.035601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12399.258327] Lustre: server umount lustre-MDT0000 complete [12401.633163] LustreError: 414622:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12401.646444] LustreError: 414622:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12403.021174] LustreError: 413503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788676747 with bad export cookie 15705996854701146902 [12403.024753] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12403.051407] LustreError: 413503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12413.498417] Lustre: server umount lustre-MDT0001 complete [12423.290828] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [12426.663258] Key type lgssc unregistered [12427.182164] LNet: 416667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12427.185201] LNetError: 416667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12427.213680] LNet: Removed LNI 192.168.202.125@tcp [12428.146586] Key type .llcrypt unregistered [12428.149442] Key type ._llcrypt unregistered [12453.715565] Key type ._llcrypt registered [12453.716724] Key type .llcrypt registered [12453.767719] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [12454.599914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12454.633405] alg: No test for adler32 (adler32-zlib) [12455.705073] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [12455.953566] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [12457.615513] Key type lgssc registered [12458.646035] Lustre: Echo OBD driver; http://www.lustre.org/ [12465.042137] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12465.056563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12480.543459] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12485.663385] LustreError: 418072:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.125@tcp: failed processing log, type 4: rc = -110 [12512.351187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12519.361949] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [12526.270528] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12534.024467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12534.281856] LustreError: 417406:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000391e1048 x1875563564239104/t0(0) o250->MGC192.168.202.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12534.851297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12534.980619] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12539.586174] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [12539.872911] LustreError: 418098:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12539.900745] LustreError: 418098:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12544.998558] LustreError: 418097:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12547.923514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12548.070567] LustreError: 419110:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12548.084497] LustreError: 419110:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12548.308978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12552.945861] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [12561.150479] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12566.758229] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12574.185105] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12574.185174] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12574.196223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12574.197540] Lustre: Skipped 1 previous similar message [12575.048516] Lustre: server umount lustre-OST0000 complete [12579.296428] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12579.299804] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12579.310749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12579.317453] Lustre: Skipped 1 previous similar message [12579.333433] Lustre: Skipped 2 previous similar messages [12581.343439] Lustre: server umount lustre-MDT0000 complete [12584.419143] LustreError: 419113:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12584.444188] LustreError: 419113:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12585.436843] LustreError: 418080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788676930 with bad export cookie 5095834381570439500 [12585.440887] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12585.452199] LustreError: 418080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12585.863548] Lustre: server umount lustre-MDT0001 complete [12596.196632] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [12598.865217] Key type lgssc unregistered [12599.157961] LNet: 421641:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12599.174434] LNetError: 421641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12599.201079] LNet: Removed LNI 192.168.202.125@tcp [12600.161318] Key type .llcrypt unregistered [12600.163511] Key type ._llcrypt unregistered [12622.527104] Key type ._llcrypt registered [12622.535601] Key type .llcrypt registered [12622.663385] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [12623.884081] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12623.942259] alg: No test for adler32 (adler32-zlib) [12624.970910] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [12625.248957] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [12626.967858] Key type lgssc registered [12628.162807] Lustre: Echo OBD driver; http://www.lustre.org/ [12636.160911] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12636.173144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12637.504675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12642.170600] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [12647.909146] LustreError: 423048:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12647.923407] LustreError: 423048:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12650.104254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12650.597542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12655.134947] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [12662.446409] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12667.937797] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12673.540253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12673.951342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12680.165529] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [12682.229346] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12686.987667] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12691.427732] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12691.434898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12691.446151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12692.448367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12692.453322] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12692.461646] Lustre: Skipped 1 previous similar message [12694.305885] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.25@tcp (stopping) [12694.316913] Lustre: Skipped 2 previous similar messages [12697.216527] Lustre: server umount lustre-MDT0000 complete [12697.575421] LustreError: 423048:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12697.587473] LustreError: 423048:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12699.568107] LustreError: 423030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.25@tcp arrived at 1788677044 with bad export cookie 366826092309897729 [12701.266856] LustreError: 423029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788677046 with bad export cookie 366826092309896833 [12701.283101] LustreError: 423029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12701.291711] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12701.667091] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12701.680255] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12701.703149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12702.692307] LustreError: 423044:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12702.717491] LustreError: 423044:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12706.724525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12706.727146] LustreError: 423048:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12706.732688] Lustre: Skipped 1 previous similar message [12706.747177] LustreError: 423048:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12707.678531] Lustre: server umount lustre-OST0000 complete [12711.844706] LustreError: 423049:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12711.855237] LustreError: 423049:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12713.320560] Lustre: server umount lustre-MDT0001 complete [12723.193123] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [12727.135090] Key type lgssc unregistered [12727.502567] LNet: 426729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12727.511702] LNetError: 426729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12727.531292] LNet: Removed LNI 192.168.202.125@tcp [12728.528516] Key type .llcrypt unregistered [12728.531660] Key type ._llcrypt unregistered [12753.820136] Key type ._llcrypt registered [12753.821524] Key type .llcrypt registered [12753.878416] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [12754.602183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12754.625951] alg: No test for adler32 (adler32-zlib) [12755.685212] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [12755.911337] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [12757.551176] Key type lgssc registered [12759.215659] Lustre: Echo OBD driver; http://www.lustre.org/ [12768.813517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12768.880976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12770.731292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12776.550946] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [12781.026118] LustreError: 428141:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12781.040700] LustreError: 428141:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12785.751819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12785.973706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12789.576838] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [12798.223529] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12803.882709] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12809.802994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12810.041559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12815.169477] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12816.014216] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [12822.847170] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12830.691695] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12830.696821] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12830.706210] Lustre: Skipped 1 previous similar message [12830.725104] Lustre: Skipped 1 previous similar message [12833.542902] Lustre: server umount lustre-MDT0001 complete [12835.820641] LustreError: 428142:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12835.843533] LustreError: 428142:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12840.928860] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12840.934589] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12840.938891] LustreError: 428141:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12840.942275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12840.942281] Lustre: Skipped 1 previous similar message [12844.075623] Lustre: server umount lustre-OST0000 complete [12846.048887] LustreError: 429209:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12848.491094] Lustre: server umount lustre-MDT0000 complete [12863.626831] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [12866.551796] Key type lgssc unregistered [12867.128704] LNet: 431827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12867.143248] LNetError: 431827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12867.164994] LNet: Removed LNI 192.168.202.125@tcp [12868.329850] Key type .llcrypt unregistered [12868.331835] Key type ._llcrypt unregistered [12894.800634] Key type ._llcrypt registered [12894.804446] Key type .llcrypt registered [12894.961741] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_hostid [12909.455267] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [12910.518454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12910.627711] alg: No test for adler32 (adler32-zlib) [12911.697611] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [12911.953166] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [12913.679186] Key type lgssc registered [12914.781096] Lustre: Echo OBD driver; http://www.lustre.org/ [12946.568631] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [12962.833181] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12962.847825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12964.020700] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12964.039523] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12964.098369] Lustre: lustre-MDT0000: new disk, initializing [12964.149582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12964.163364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12968.884153] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [12983.831321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12984.106575] Lustre: lustre-OST0000: new disk, initializing [12984.109963] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12984.115522] Lustre: Skipped 1 previous similar message [12984.139947] Lustre: 436858:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12984.229639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12990.323418] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [12990.573452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12990.587303] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12990.618523] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12997.816565] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13011.570749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13011.685543] Lustre: 435840:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13011.738906] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13011.807278] Lustre: lustre-MDT0001: new disk, initializing [13011.904978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13011.938326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13011.950136] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13015.672382] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [13024.794871] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13024.987617] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13029.200125] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13029.392296] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13034.993646] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13035.003804] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13035.020221] Lustre: Skipped 1 previous similar message [13035.046241] Lustre: Skipped 1 previous similar message [13038.869630] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.25@tcp (stopping) [13040.101698] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13040.112297] Lustre: Skipped 1 previous similar message [13043.962330] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.25@tcp (stopping) [13048.287329] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13048.483987] Lustre: server umount lustre-MDT0001 complete [13055.464893] LustreError: 436866:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13055.491828] LustreError: 436866:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13060.590869] LustreError: 435850:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13060.615120] LustreError: 435850:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13065.709790] LustreError: 436866:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13065.744064] LustreError: 436866:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13067.232331] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13067.241734] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13067.266986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13067.285027] Lustre: Skipped 2 previous similar messages [13070.821852] LustreError: 435851:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13073.450542] Lustre: server umount lustre-OST0000 complete [13075.955313] LustreError: 435850:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13085.770639] Lustre: server umount lustre-MDT0000 complete [13098.824453] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [13101.398328] Key type lgssc unregistered [13101.586755] LNet: 440604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13101.591130] LNetError: 440604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13101.599816] LNet: Removed LNI 192.168.202.125@tcp [13102.131662] Key type .llcrypt unregistered [13102.141778] Key type ._llcrypt unregistered [13135.547482] Key type ._llcrypt registered [13135.548885] Key type .llcrypt registered [13135.613133] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [13135.894702] Key type .llcrypt unregistered [13135.897780] Key type ._llcrypt unregistered [13150.462676] Key type ._llcrypt registered [13150.464739] Key type .llcrypt registered [13150.585330] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [13151.643954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13151.727830] alg: No test for adler32 (adler32-zlib) [13152.750738] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [13152.998717] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [13154.623171] Key type lgssc registered [13155.639232] Lustre: Echo OBD driver; http://www.lustre.org/ [13180.703535] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13180.729473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13181.959640] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13182.014862] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13182.086309] Lustre: lustre-MDT0000: new disk, initializing [13182.157398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13182.176104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13185.236087] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [13194.899264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13195.046545] Lustre: 443971:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13195.113548] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13195.117611] Lustre: Skipped 1 previous similar message [13195.202305] Lustre: lustre-MDT0001: new disk, initializing [13195.313338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13195.400072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13195.421530] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13200.747748] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [13211.078770] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13216.498841] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13222.717378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13223.136252] Lustre: lustre-OST0000: new disk, initializing [13223.144803] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13223.153350] Lustre: 445938:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13223.234230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13228.824545] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [13230.156417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13230.170148] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13230.295817] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13237.328740] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13245.921811] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13245.927417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13245.948841] Lustre: Skipped 1 previous similar message [13245.970238] Lustre: Skipped 1 previous similar message [13250.190197] Lustre: server umount lustre-MDT0000 complete [13255.105192] LustreError: 445937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788677600 with bad export cookie 17939823100985401740 [13255.105961] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13255.117126] LustreError: 445937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13255.503263] Lustre: server umount lustre-MDT0001 complete [13269.719678] Lustre: server umount lustre-OST0000 complete [13279.617360] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 02:53:44 (1788677624) [13287.805184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13288.426753] LustreError: 447966:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13288.453023] LustreError: 447966:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13288.500081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13292.980135] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [13293.543194] LustreError: 447967:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13298.657733] LustreError: 447966:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13300.610989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13301.113156] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13305.606585] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [13313.692639] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13319.368599] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13339.620474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13339.797125] Lustre: Found index 0 for lustre-OST0000, updating log [13339.944425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13346.563857] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [13353.827415] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13359.422450] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13364.711696] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13364.712412] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13364.720196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13364.744730] Lustre: Skipped 2 previous similar messages [13365.819585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13365.828342] Lustre: Skipped 1 previous similar message [13366.794889] Lustre: server umount lustre-OST0000 complete [13383.136230] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13383.143689] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13383.153688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13383.158853] Lustre: Skipped 1 previous similar message [13387.232122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13387.249322] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13389.126386] Lustre: server umount lustre-MDT0000 complete [13391.398095] LustreError: 447967:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13393.002687] LustreError: 447948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788677737 with bad export cookie 17939823100985403133 [13393.021586] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13393.414503] Lustre: server umount lustre-MDT0001 complete [13402.563112] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 02:55:47 (1788677747) [13403.875672] LNet: 1 local NIs in recovery (showing 1): 192.168.202.125@tcp [14021.710061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14022.193512] LustreError: 454391:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14022.212087] LustreError: 454391:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [14022.272292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14026.781230] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [14027.746768] LustreError: 454392:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14032.868691] LustreError: 454391:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14034.983808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14035.250207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14039.329982] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [14046.457900] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14052.005442] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14071.789571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14072.154352] Lustre: Found index 0 for lustre-OST0000, updating log [14072.466682] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14079.327727] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [14081.760029] LNet: 1 local NIs in recovery (showing 1): 192.168.202.125@tcp [14087.582655] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14089.772376] Lustre: server umount lustre-OST0000 complete [14093.555723] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14093.568713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14093.572361] Lustre: Skipped 1 previous similar message [14093.586586] Lustre: Skipped 2 previous similar messages [14094.628923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14094.633412] Lustre: Skipped 1 previous similar message [14096.674197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14096.683809] Lustre: Skipped 3 previous similar messages [14100.046522] Lustre: server umount lustre-MDT0000 complete [14100.770804] LustreError: 454388:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14100.783737] LustreError: 454388:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14103.845749] LustreError: 454372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788678448 with bad export cookie 17939823100985404218 [14103.851417] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14103.857843] LustreError: 454372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [14104.241767] Lustre: server umount lustre-MDT0001 complete [14133.910867] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_hostid [14142.712421] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [14176.535627] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 03:08:41 (1788678521) [14195.112251] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [14197.295490] Key type lgssc unregistered [14197.560504] LNet: 461006:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14197.568148] LNetError: 461006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14325.671847] LNet: Removed LNI 192.168.202.125@tcp [14326.767561] Key type .llcrypt unregistered [14326.777110] Key type ._llcrypt unregistered [14343.435397] Key type ._llcrypt registered [14343.438254] Key type .llcrypt registered [14343.530420] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [14344.405125] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14344.449857] alg: No test for adler32 (adler32-zlib) [14345.625932] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [14345.854532] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [14347.519172] Key type lgssc registered [14348.672276] Lustre: Echo OBD driver; http://www.lustre.org/ [14352.946928] LNet: Added LNI 192.168.202.125@tcp100 [8/256/0/180] [14353.759156] LNet: Added LNI 192.168.202.125@tcp101 [8/256/0/180] [14354.686235] LNet: Added LNI 192.168.202.125@tcp102 [8/256/0/180] [14355.549988] LNet: Added LNI 192.168.202.125@tcp103 [8/256/0/180] [14356.491015] LNet: Added LNI 192.168.202.125@tcp104 [8/256/0/180] [14357.578047] LNet: Added LNI 192.168.202.125@tcp105 [8/256/0/180] [14358.489971] LNet: Added LNI 192.168.202.125@tcp106 [8/256/0/180] [14359.212560] LNet: Added LNI 192.168.202.125@tcp107 [8/256/0/180] [14359.986834] LNet: Added LNI 192.168.202.125@tcp108 [8/256/0/180] [14360.790447] LNet: Added LNI 192.168.202.125@tcp109 [8/256/0/180] [14361.529563] LNet: Added LNI 192.168.202.125@tcp110 [8/256/0/180] [14362.223491] LNet: Added LNI 192.168.202.125@tcp111 [8/256/0/180] [14363.020714] LNet: Added LNI 192.168.202.125@tcp112 [8/256/0/180] [14363.798639] LNet: Added LNI 192.168.202.125@tcp113 [8/256/0/180] [14364.627674] LNet: Added LNI 192.168.202.125@tcp114 [8/256/0/180] [14365.592329] LNet: Added LNI 192.168.202.125@tcp115 [8/256/0/180] [14366.489491] LNet: Added LNI 192.168.202.125@tcp116 [8/256/0/180] [14367.383374] LNet: Added LNI 192.168.202.125@tcp117 [8/256/0/180] [14368.303470] LNet: Added LNI 192.168.202.125@tcp118 [8/256/0/180] [14369.290297] LNet: Added LNI 192.168.202.125@tcp119 [8/256/0/180] [14370.248215] LNet: Added LNI 192.168.202.125@tcp120 [8/256/0/180] [14371.103737] LNet: Added LNI 192.168.202.125@tcp121 [8/256/0/180] [14372.175835] LNet: Added LNI 192.168.202.125@tcp122 [8/256/0/180] [14372.884142] LNet: Added LNI 192.168.202.125@tcp123 [8/256/0/180] [14373.693208] LNet: Added LNI 192.168.202.125@tcp124 [8/256/0/180] [14374.583821] LNet: Added LNI 192.168.202.125@tcp125 [8/256/0/180] [14375.460037] LNet: Added LNI 192.168.202.125@tcp126 [8/256/0/180] [14376.334208] LNet: Added LNI 192.168.202.125@tcp127 [8/256/0/180] [14377.169937] LNet: Added LNI 192.168.202.125@tcp128 [8/256/0/180] [14378.195879] LNet: Added LNI 192.168.202.125@tcp129 [8/256/0/180] [14379.018258] LNet: Added LNI 192.168.202.125@tcp130 [8/256/0/180] [14379.899579] LNet: Added LNI 192.168.202.125@tcp131 [8/256/0/180] [14380.610371] LNet: Added LNI 192.168.202.125@tcp132 [8/256/0/180] [14381.458334] LNet: Added LNI 192.168.202.125@tcp133 [8/256/0/180] [14382.410890] LNet: Added LNI 192.168.202.125@tcp134 [8/256/0/180] [14383.235046] LNet: Added LNI 192.168.202.125@tcp135 [8/256/0/180] [14390.276285] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14390.315954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14391.453522] Lustre: 464035:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.125@tcp: can't add NID 192.168.202.125@tcp130: rc = -75 [14391.575109] LNetError: 464062:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.125 [14391.586166] LNetError: Protocol error connecting to 192.168.202.125@tcp117 at host 192.168.202.125:988: is it running a compatible version of Lustre? [14391.603222] LNetError: Skipped 9 previous similar messages [14391.683948] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14391.725338] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14391.841677] Lustre: lustre-MDT0000: new disk, initializing [14391.959773] Lustre: 464092:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.125@tcp: can't add NID 192.168.202.125@tcp130: rc = -75 [14391.968112] LNet: 7 local NIs in recovery (showing 7): 192.168.202.125@tcp123, 192.168.202.125@tcp125, 192.168.202.125@tcp126, 192.168.202.125@tcp127, 192.168.202.125@tcp124, 192.168.202.125@tcp129, 192.168.202.125@tcp128 [14391.971085] Lustre: 464092:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14392.010723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14392.029723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14392.616333] LNetError: 464056:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.125 [14392.645066] LNetError: 464056:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [14392.654939] LNetError: Protocol error connecting to 192.168.202.125@tcp107 at host 192.168.202.125:988: is it running a compatible version of Lustre? [14394.018216] LNetError: 464065:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.125 [14394.033254] LNetError: 464065:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [14394.041149] LNetError: Protocol error connecting to 192.168.202.125@tcp123 at host 192.168.202.125:988: is it running a compatible version of Lustre? [14394.045498] LNetError: Skipped 7 previous similar messages [14397.125823] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [14398.116490] LNetError: 464069:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.125 [14398.121816] LNetError: 464069:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 5 previous similar messages [14398.141714] LNetError: Protocol error connecting to 192.168.202.125@tcp123 at host 192.168.202.125:988: is it running a compatible version of Lustre? [14398.157765] LNetError: Skipped 3 previous similar messages [14406.312207] LNetError: 461494:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.125 [14406.324837] LNetError: 461494:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [14406.332256] LNetError: Protocol error connecting to 192.168.202.125@tcp123 at host 192.168.202.125:988: is it running a compatible version of Lustre? [14407.688427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14407.826563] Lustre: 464052:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14407.897550] Lustre: 464856:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.125@tcp: can't add NID 192.168.202.125@tcp130: rc = -75 [14407.907437] Lustre: 464856:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14407.923723] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14407.928995] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [14407.930469] Lustre: Skipped 1 previous similar message [14407.997687] Lustre: lustre-MDT0001: new disk, initializing [14408.146297] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14408.177621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14408.183837] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14412.172843] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [14422.688857] LNetError: 461491:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.125 [14422.700230] LNetError: Protocol error connecting to 192.168.202.125@tcp123 at host 192.168.202.125:988: is it running a compatible version of Lustre? [14422.825295] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14428.513824] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14435.369161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14435.750720] Lustre: lustre-OST0000: new disk, initializing [14435.754405] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14435.761976] Lustre: 466028:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14435.877710] Lustre: 466033:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.125@tcp: can't add NID 192.168.202.125@tcp130: rc = -75 [14435.884779] Lustre: 466033:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [14435.936759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14439.139731] LNetError: 464065:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.125 [14439.153351] LNetError: 464065:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 15 previous similar messages [14439.171284] LNetError: Protocol error connecting to 192.168.202.125@tcp128 at host 192.168.202.125:988: is it running a compatible version of Lustre? [14439.185963] LNetError: Skipped 15 previous similar messages [14441.482930] Lustre: 466341:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.125@tcp: can't add NID 192.168.202.125@tcp130: rc = -75 [14441.487961] Lustre: 466341:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14441.542241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14441.549071] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14441.737439] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14441.853812] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [14450.530650] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14451.361523] LNet: 1 local NIs in recovery (showing 1): 192.168.202.125@tcp123 [14462.443431] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14462.449192] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14462.465622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14462.480489] Lustre: Skipped 1 previous similar message [14467.556313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14467.562381] Lustre: Skipped 1 previous similar message [14471.904666] LNetError: 461491:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.125 [14471.917579] LNetError: 461491:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 24 previous similar messages [14471.925719] LNetError: Protocol error connecting to 192.168.202.125@tcp127 at host 192.168.202.125:988: is it running a compatible version of Lustre? [14471.936243] LNetError: Skipped 24 previous similar messages [14472.159400] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14472.389608] Lustre: server umount lustre-OST0000 complete [14477.811824] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14477.839335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14477.843733] Lustre: Skipped 1 previous similar message [14482.446506] Lustre: server umount lustre-MDT0000 complete [14485.616472] LustreError: 464044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788678830 with bad export cookie 7357275388913489367 [14485.617055] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14485.627899] LustreError: 464044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14486.179439] Lustre: server umount lustre-MDT0001 complete [14497.116682] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [14500.328569] Key type lgssc unregistered [14500.780840] LNet: 467941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14500.788565] LNetError: 467941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14500.796502] LNet: Removed LNI 192.168.202.125@tcp [14501.855520] LNet: Removed LNI 192.168.202.125@tcp100 [14501.868638] LNetError: 467941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14501.887169] LNetError: 467941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14501.914850] LNet: Removed LNI 192.168.202.125@tcp101 [14502.943357] LNet: Removed LNI 192.168.202.125@tcp102 [14502.945503] LNetError: 467941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14502.957233] LNetError: 467941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14502.969386] LNet: Removed LNI 192.168.202.125@tcp103 [14504.031200] LNet: Removed LNI 192.168.202.125@tcp104 [14505.055658] LNet: Removed LNI 192.168.202.125@tcp105 [14505.057877] LNetError: 467941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14505.070702] LNetError: 467941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14505.082773] LNet: Removed LNI 192.168.202.125@tcp106 [14506.143215] LNet: Removed LNI 192.168.202.125@tcp107 [14507.167159] LNet: Removed LNI 192.168.202.125@tcp108 [14508.192359] LNet: Removed LNI 192.168.202.125@tcp109 [14509.215722] LNet: Removed LNI 192.168.202.125@tcp110 [14509.224663] LNetError: 467941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14509.240796] LNetError: 467941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14509.251913] LNet: Removed LNI 192.168.202.125@tcp111 [14510.303989] LNet: Removed LNI 192.168.202.125@tcp112 [14511.327395] LNet: Removed LNI 192.168.202.125@tcp113 [14512.351318] LNet: Removed LNI 192.168.202.125@tcp114 [14513.375594] LNet: Removed LNI 192.168.202.125@tcp115 [14514.399688] LNet: Removed LNI 192.168.202.125@tcp116 [14515.423250] LNet: Removed LNI 192.168.202.125@tcp117 [14516.447474] LNet: Removed LNI 192.168.202.125@tcp118 [14517.472743] LNet: Removed LNI 192.168.202.125@tcp119 [14517.496802] LNetError: 467941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14517.508647] LNetError: 467941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14517.532747] LNet: Removed LNI 192.168.202.125@tcp120 [14518.559298] LNet: Removed LNI 192.168.202.125@tcp121 [14519.583214] LNet: Removed LNI 192.168.202.125@tcp122 [14519.585389] LNet: Removed LNI 192.168.202.125@tcp123 [14520.608026] LNet: Removed LNI 192.168.202.125@tcp124 [14521.631309] LNet: Removed LNI 192.168.202.125@tcp125 [14522.655208] LNet: Removed LNI 192.168.202.125@tcp126 [14523.679263] LNet: Removed LNI 192.168.202.125@tcp127 [14524.703943] LNet: Removed LNI 192.168.202.125@tcp128 [14525.727152] LNet: Removed LNI 192.168.202.125@tcp129 [14525.738941] LNet: Removed LNI 192.168.202.125@tcp130 [14525.751330] LNet: Removed LNI 192.168.202.125@tcp131 [14525.760946] LNet: Removed LNI 192.168.202.125@tcp132 [14525.773842] LNet: Removed LNI 192.168.202.125@tcp133 [14525.779101] LNet: Removed LNI 192.168.202.125@tcp134 [14525.807088] LNet: Removed LNI 192.168.202.125@tcp135 [14526.746940] Key type .llcrypt unregistered [14526.753896] Key type ._llcrypt unregistered [14553.202191] Key type ._llcrypt registered [14553.204467] Key type .llcrypt registered [14553.412505] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_hostid [14571.015332] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [14571.976743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14571.994030] alg: No test for adler32 (adler32-zlib) [14573.012464] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [14573.265655] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [14575.047226] Key type lgssc registered [14576.164338] Lustre: Echo OBD driver; http://www.lustre.org/ [14622.477967] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 03:16:06 (1788678966) [14641.711257] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [14644.687071] Key type lgssc unregistered [14645.083527] LNet: 472027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14645.097345] LNetError: 472027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14645.125284] LNet: Removed LNI 192.168.202.125@tcp [14645.999528] Key type .llcrypt unregistered [14646.001122] Key type ._llcrypt unregistered [14662.997053] Key type ._llcrypt registered [14662.999188] Key type .llcrypt registered [14663.107151] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [14664.490774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14664.756397] alg: No test for adler32 (adler32-zlib) [14666.066907] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [14666.486429] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [14668.232533] Key type lgssc registered [14669.588794] Lustre: Echo OBD driver; http://www.lustre.org/ [14679.352872] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14679.430809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14680.819171] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14680.847926] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14680.901943] Lustre: lustre-MDT0000: new disk, initializing [14680.968175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14680.985871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14685.904924] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [14696.399253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14696.567958] Lustre: 473236:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14696.624875] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14696.628149] Lustre: Skipped 1 previous similar message [14696.710233] Lustre: lustre-MDT0001: new disk, initializing [14696.837793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14696.909106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14696.931142] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14701.926284] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [14711.652855] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14717.696205] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14724.340658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14724.724629] Lustre: lustre-OST0000: new disk, initializing [14724.728462] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14724.744585] Lustre: 475205:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14724.830671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14730.325191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14730.344183] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14730.469898] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14731.377431] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [14741.199775] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14744.894060] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing lnet_if_list [14750.536898] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14750.557528] LNet: Added LNI 192.168.202.125@tcp10 [8/256/0/180] [14756.213963] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14756.269760] LNet: Added LNI 192.168.202.125@tcp11 [8/256/0/180] [14761.682908] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14761.717857] LNet: Added LNI 192.168.202.125@tcp12 [8/256/0/180] [14767.164192] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14767.213080] LNet: Added LNI 192.168.202.125@tcp13 [8/256/0/180] [14774.155380] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14774.209641] LNet: Added LNI 192.168.202.125@tcp14 [8/256/0/180] [14779.601845] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14779.636469] LNet: Added LNI 192.168.202.125@tcp15 [8/256/0/180] [14784.977961] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14784.990336] LNet: Added LNI 192.168.202.125@tcp16 [8/256/0/180] [14789.258807] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14789.293532] LNet: Added LNI 192.168.202.125@tcp17 [8/256/0/180] [14793.275749] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14793.297635] LNet: Added LNI 192.168.202.125@tcp18 [8/256/0/180] [14798.155939] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14798.176109] LNet: Added LNI 192.168.202.125@tcp19 [8/256/0/180] [14803.837142] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14803.873122] LNet: Added LNI 192.168.202.125@tcp20 [8/256/0/180] [14809.506362] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14809.536470] LNet: Added LNI 192.168.202.125@tcp21 [8/256/0/180] [14815.413969] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14815.445542] LNet: Added LNI 192.168.202.125@tcp22 [8/256/0/180] [14821.651128] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14821.688433] LNet: Added LNI 192.168.202.125@tcp23 [8/256/0/180] [14826.423996] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14826.449732] LNet: Added LNI 192.168.202.125@tcp24 [8/256/0/180] [14832.993937] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14833.034831] LNet: Added LNI 192.168.202.125@tcp25 [8/256/0/180] [14842.134974] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14842.190495] LNet: Added LNI 192.168.202.125@tcp26 [8/256/0/180] [14850.006409] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14850.026686] LNet: Added LNI 192.168.202.125@tcp27 [8/256/0/180] [14858.088945] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14858.143369] LNet: Added LNI 192.168.202.125@tcp28 [8/256/0/180] [14865.880772] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14865.914558] LNet: Added LNI 192.168.202.125@tcp29 [8/256/0/180] [14873.452700] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14873.488231] LNet: Added LNI 192.168.202.125@tcp30 [8/256/0/180] [14880.506504] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14880.579267] LNet: Added LNI 192.168.202.125@tcp31 [8/256/0/180] [14887.931155] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14887.961722] LNet: Added LNI 192.168.202.125@tcp32 [8/256/0/180] [14895.543555] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14895.579440] LNet: Added LNI 192.168.202.125@tcp33 [8/256/0/180] [14902.980898] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14903.002837] LNet: Added LNI 192.168.202.125@tcp34 [8/256/0/180] [14911.114718] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14911.165622] LNet: Added LNI 192.168.202.125@tcp35 [8/256/0/180] [14919.234513] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14919.315796] LNet: Added LNI 192.168.202.125@tcp36 [8/256/0/180] [14926.122032] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14926.188891] LNet: Added LNI 192.168.202.125@tcp37 [8/256/0/180] [14933.197813] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14933.245245] LNet: Added LNI 192.168.202.125@tcp38 [8/256/0/180] [14941.670796] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14941.702569] LNet: Added LNI 192.168.202.125@tcp39 [8/256/0/180] [14951.143324] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14951.190836] LNet: Added LNI 192.168.202.125@tcp40 [8/256/0/180] [14959.676781] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14959.711700] LNet: Added LNI 192.168.202.125@tcp41 [8/256/0/180] [14968.202205] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14968.227374] LNet: Added LNI 192.168.202.125@tcp42 [8/256/0/180] [14976.741532] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14976.787819] LNet: Added LNI 192.168.202.125@tcp43 [8/256/0/180] [14985.549234] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14985.578894] LNet: Added LNI 192.168.202.125@tcp44 [8/256/0/180] [14992.930136] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14992.963292] LNet: Added LNI 192.168.202.125@tcp45 [8/256/0/180] [15001.557638] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15001.578049] LNet: Added LNI 192.168.202.125@tcp46 [8/256/0/180] [15009.022386] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15009.049427] LNet: Added LNI 192.168.202.125@tcp47 [8/256/0/180] [15017.465310] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15017.511873] LNet: Added LNI 192.168.202.125@tcp48 [8/256/0/180] [15025.604762] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15025.618653] LNet: Added LNI 192.168.202.125@tcp49 [8/256/0/180] [15034.617619] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15034.638316] LNet: Added LNI 192.168.202.125@tcp50 [8/256/0/180] [15058.402118] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15058.406630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15058.416896] Lustre: Skipped 1 previous similar message [15058.426185] Lustre: Skipped 2 previous similar messages [15062.723721] Lustre: server umount lustre-MDT0000 complete [15066.157859] LustreError: 473230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788679411 with bad export cookie 14813164586709843976 [15066.166561] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15066.175883] LustreError: 473230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15066.533143] Lustre: server umount lustre-MDT0001 complete [15084.968053] Lustre: 472507:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788679413/real 1788679413] req@000000009e54f286 x1875565882818944/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788679429 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15085.005069] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15085.027097] Lustre: Skipped 1 previous similar message [15085.814407] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [15087.071108] Lustre: 472509:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788679416/real 1788679416] req@000000001733a49e x1875565882819072/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788679432 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15089.183079] Lustre: 472509:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788679418/real 1788679418] req@00000000ffb83c0b x1875565882819200/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788679434 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15092.128683] Lustre: 472507:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788679421/real 1788679421] req@00000000a93107b4 x1875565882819456/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788679437 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15094.173855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15097.311423] Lustre: 472506:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788679426/real 1788679426] req@000000009b568b0c x1875565882819584/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788679442 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15111.913217] LustreError: 475199:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15111.937738] LustreError: 475199:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [15111.987278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15114.027791] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15114.031436] LustreError: 475200:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15117.884878] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [15119.331238] LustreError: 475199:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15124.456114] LustreError: 475200:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15127.116991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15127.588094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15129.645883] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15133.388734] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [15141.992693] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15148.607603] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15170.535927] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15170.553800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15170.567501] Lustre: Skipped 1 previous similar message [15175.694206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15175.703537] Lustre: Skipped 1 previous similar message [15180.772214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15180.789678] Lustre: Skipped 1 previous similar message [15184.351133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15184.607617] Lustre: server umount lustre-OST0000 complete [15191.008746] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15191.031391] Lustre: Skipped 2 previous similar messages [15191.042387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15191.056631] Lustre: Skipped 1 previous similar message [15196.128318] LustreError: 499454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15196.144986] LustreError: 499454:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15196.203425] Lustre: server umount lustre-MDT0000 complete [15201.109045] LustreError: 473229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788679546 with bad export cookie 14813164586709846636 [15201.112945] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15201.129343] LustreError: 473229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15201.563913] Lustre: server umount lustre-MDT0001 complete [15212.108429] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [15215.808489] Key type lgssc unregistered [15216.095834] LNet: 502250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15216.103522] LNetError: 502250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15217.119269] LNet: Removed LNI 192.168.202.125@tcp [15217.122164] LNetError: 502250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15217.133962] LNet: Removed LNI 192.168.202.125@tcp10 [15217.152468] LNet: Removed LNI 192.168.202.125@tcp11 [15217.159310] LNet: Removed LNI 192.168.202.125@tcp12 [15217.161557] LNet: Removed LNI 192.168.202.125@tcp13 [15217.164093] LNet: Removed LNI 192.168.202.125@tcp14 [15217.169526] LNet: Removed LNI 192.168.202.125@tcp15 [15217.174229] LNet: Removed LNI 192.168.202.125@tcp16 [15217.177182] LNet: Removed LNI 192.168.202.125@tcp17 [15217.181580] LNet: Removed LNI 192.168.202.125@tcp18 [15217.189463] LNet: Removed LNI 192.168.202.125@tcp19 [15217.193288] LNet: Removed LNI 192.168.202.125@tcp20 [15217.201706] LNet: Removed LNI 192.168.202.125@tcp21 [15217.205306] LNet: Removed LNI 192.168.202.125@tcp22 [15217.214680] LNet: Removed LNI 192.168.202.125@tcp23 [15217.222843] LNet: Removed LNI 192.168.202.125@tcp24 [15217.229852] LNet: Removed LNI 192.168.202.125@tcp25 [15217.232916] LNet: Removed LNI 192.168.202.125@tcp26 [15217.235935] LNet: Removed LNI 192.168.202.125@tcp27 [15217.238387] LNet: Removed LNI 192.168.202.125@tcp28 [15217.248189] LNet: Removed LNI 192.168.202.125@tcp29 [15217.252185] LNet: Removed LNI 192.168.202.125@tcp30 [15217.260159] LNet: Removed LNI 192.168.202.125@tcp31 [15217.268926] LNet: Removed LNI 192.168.202.125@tcp32 [15217.270894] LNet: Removed LNI 192.168.202.125@tcp33 [15217.275561] LNet: Removed LNI 192.168.202.125@tcp34 [15217.280136] LNet: Removed LNI 192.168.202.125@tcp35 [15217.285495] LNet: Removed LNI 192.168.202.125@tcp36 [15217.288108] LNet: Removed LNI 192.168.202.125@tcp37 [15217.291313] LNet: Removed LNI 192.168.202.125@tcp38 [15217.292713] LNet: Removed LNI 192.168.202.125@tcp39 [15217.303486] LNet: Removed LNI 192.168.202.125@tcp40 [15217.308401] LNet: Removed LNI 192.168.202.125@tcp41 [15217.316592] LNet: Removed LNI 192.168.202.125@tcp42 [15217.318184] LNet: Removed LNI 192.168.202.125@tcp43 [15217.319409] LNet: Removed LNI 192.168.202.125@tcp44 [15217.320636] LNet: Removed LNI 192.168.202.125@tcp45 [15217.321993] LNet: Removed LNI 192.168.202.125@tcp46 [15217.323224] LNet: Removed LNI 192.168.202.125@tcp47 [15217.324874] LNet: Removed LNI 192.168.202.125@tcp48 [15217.328786] LNet: Removed LNI 192.168.202.125@tcp49 [15217.342919] LNet: Removed LNI 192.168.202.125@tcp50 [15218.527692] Key type .llcrypt unregistered [15218.529086] Key type ._llcrypt unregistered [15240.805469] Key type ._llcrypt registered [15240.808330] Key type .llcrypt registered [15240.925610] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_hostid [15255.838325] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [15256.948746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15257.049464] alg: No test for adler32 (adler32-zlib) [15258.365380] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [15258.635553] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [15260.431183] Key type lgssc registered [15261.654355] Lustre: Echo OBD driver; http://www.lustre.org/ [15296.510814] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15296.565925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15297.941641] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15298.009760] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15298.161913] Lustre: lustre-MDT0000: new disk, initializing [15298.310054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15298.330081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15303.283122] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [15314.013900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15314.125545] Lustre: 505443:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15314.174674] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15314.178552] Lustre: Skipped 1 previous similar message [15314.242705] Lustre: lustre-MDT0001: new disk, initializing [15314.302881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15314.360221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15314.371245] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15319.369521] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [15328.295125] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15334.285584] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15342.917559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15343.038929] Lustre: *** cfs_fail_loc=90f, val=0*** [15343.083469] Lustre: lustre-OST0000: new disk, initializing [15343.085772] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15343.090917] Lustre: 507459:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15343.140804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15350.349205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15350.366297] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15350.470259] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15351.514560] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [15361.572536] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15371.240399] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15371.245722] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15371.256312] LustreError: Skipped 1 previous similar message [15371.277875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15374.313309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15374.318466] Lustre: Skipped 1 previous similar message [15375.649817] Lustre: server umount lustre-OST0000 complete [15380.966271] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15380.988916] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15381.009397] Lustre: Skipped 1 previous similar message [15381.017710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15384.546690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15384.547047] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15386.008553] Lustre: server umount lustre-MDT0000 complete [15389.712294] LustreError: 505435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788679734 with bad export cookie 2205936503801336283 [15389.721817] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15389.725946] LustreError: 505435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15390.056947] Lustre: server umount lustre-MDT0001 complete [15401.130418] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [15404.468356] Key type lgssc unregistered [15404.834887] LNet: 509369:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15404.847919] LNetError: 509369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15404.866725] LNet: Removed LNI 192.168.202.125@tcp [15405.957376] Key type .llcrypt unregistered [15405.960271] Key type ._llcrypt unregistered [15427.213724] Key type ._llcrypt registered [15427.216802] Key type .llcrypt registered [15427.375088] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [15428.581987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15428.756048] alg: No test for adler32 (adler32-zlib) [15430.128184] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [15430.360950] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [15432.031286] Key type lgssc registered [15433.561367] Lustre: Echo OBD driver; http://www.lustre.org/ [15443.245461] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15443.281661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15444.983525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15450.837222] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [15454.189811] LustreError: 510635:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15454.205294] LustreError: 510635:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15459.300186] LustreError: 510634:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15460.440939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15460.924360] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15466.210813] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [15474.964067] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15481.222593] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15488.512939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15488.946865] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15495.877648] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [15504.091339] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15516.615658] loop0: detected capacity change from 0 to 204800000 [15516.867436] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15516.893054] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15516.911805] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15523.504299] loop0: detected capacity change from 0 to 204800000 [15523.760047] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15523.780857] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15523.789929] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15529.044523] loop0: detected capacity change from 0 to 204800000 [15534.844734] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15535.048909] Lustre: fs15246-OST0000: new disk, initializing [15535.052962] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15535.057882] Lustre: 514228:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15535.153095] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15541.502514] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [15551.987993] loop1: detected capacity change from 0 to 204800000 [15558.080914] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15558.222835] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15558.271293] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15558.330481] Lustre: fs15246-MDT0000: new disk, initializing [15558.494424] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15558.521276] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15558.686906] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15559.778218] Lustre: 515351:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000415f7f10 x1875566683799168/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788679920 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15563.285291] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [15568.875443] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15568.894294] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15568.931093] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15586.789390] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15586.823773] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15591.869066] Lustre: server umount fs15246-MDT0000 complete [15591.927661] LustreError: 510629:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15597.042904] LustreError: 511600:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15601.467037] Lustre: server umount fs15246-OST0000 complete [15617.515370] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15617.520072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15617.546965] Lustre: Skipped 1 previous similar message [15620.964977] Lustre: server umount lustre-OST0000 complete [15627.748102] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15627.767832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15627.776552] Lustre: Skipped 2 previous similar messages [15632.043595] Lustre: server umount lustre-MDT0000 complete [15632.873242] LustreError: 510635:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15632.902462] LustreError: 510635:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15636.224588] LustreError: 510617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788679981 with bad export cookie 11141616770254261913 [15636.230450] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15636.239203] LustreError: 510617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15636.702395] Lustre: server umount lustre-MDT0001 complete [15649.538714] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [15652.789438] Key type lgssc unregistered [15653.086198] LNet: 518445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15653.098271] LNetError: 518445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15653.117309] LNet: Removed LNI 192.168.202.125@tcp [15654.114318] Key type .llcrypt unregistered [15654.117519] Key type ._llcrypt unregistered [15674.040301] Key type ._llcrypt registered [15674.044687] Key type .llcrypt registered [15674.183941] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [15675.707334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15675.864801] alg: No test for adler32 (adler32-zlib) [15677.174579] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [15677.584478] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [15679.383207] Key type lgssc registered [15681.042695] Lustre: Echo OBD driver; http://www.lustre.org/ [15722.432369] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_hostid [15733.142128] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [15768.185703] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15768.233711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15769.661031] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15769.698660] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15769.794299] Lustre: lustre-MDT0000: new disk, initializing [15769.920675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15769.947887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15774.256581] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [15784.287719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15784.399284] Lustre: 522909:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15784.476901] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15784.482442] Lustre: Skipped 1 previous similar message [15784.542165] Lustre: lustre-MDT0001: new disk, initializing [15784.632681] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15784.663181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15784.678698] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15789.238585] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [15799.178921] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15805.405755] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15812.287198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15812.541801] Lustre: lustre-OST0000: new disk, initializing [15812.546154] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15812.556738] Lustre: 524876:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15812.668703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15818.988547] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [15821.890464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15821.897596] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15822.011907] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15829.603485] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15834.989378] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15835.227391] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15839.817539] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15840.048834] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15842.786883] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15842.801147] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15842.826075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15842.835160] Lustre: Skipped 1 previous similar message [15847.909847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15847.921022] Lustre: Skipped 1 previous similar message [15848.085251] Lustre: server umount lustre-OST0000 complete [15853.035034] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15853.047280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15853.056980] Lustre: Skipped 2 previous similar messages [15858.149569] LustreError: 522916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15858.176622] LustreError: 522916:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [15860.015733] Lustre: server umount lustre-MDT0000 complete [15863.280900] LustreError: 522917:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15863.300117] LustreError: 522917:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15863.682330] LustreError: 522900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788680208 with bad export cookie 16825510119088682585 [15863.712787] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15863.979381] Lustre: server umount lustre-MDT0001 complete [15891.373582] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_hostid [15902.779547] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [15934.427623] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15676 sec ========= 03:37:58 (1788680278) [15936.045076] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:38:00 (1788680280) === [15940.022731] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:38:04 (1788680284) === [15980.485162] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [15983.235546] Key type lgssc unregistered [15983.565872] LNet: 531518:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15983.576859] LNetError: 531518:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15983.609825] LNet: Removed LNI 192.168.202.125@tcp [15984.418469] Key type .llcrypt unregistered [15984.420232] Key type ._llcrypt unregistered