[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 579092744 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002329] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004009] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.007017] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.008006] pid_max: default: 32768 minimum: 301 [ 0.009124] LSM: Security Framework initializing [ 0.010037] Yama: becoming mindful. [ 0.011027] SELinux: Initializing. [ 0.012048] *** VALIDATE selinux *** [ 0.020064] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024495] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025142] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026081] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027093] *** VALIDATE tmpfs *** [ 0.028390] *** VALIDATE proc *** [ 0.029199] *** VALIDATE cgroup *** [ 0.030005] *** VALIDATE cgroup2 *** [ 0.031217] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032125] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034024] Spectre V2 : User space: Vulnerable [ 0.035006] Speculative Store Bypass: Vulnerable [ 0.037781] debug: unmapping init [mem 0xffffffff9c059000-0xffffffff9c060fff] [ 0.039893] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.040561] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.041016] ... version: 2 [ 0.042008] ... bit width: 48 [ 0.043007] ... generic registers: 4 [ 0.044009] ... value mask: 0000ffffffffffff [ 0.045008] ... max period: 00007fffffffffff [ 0.046009] ... fixed-purpose events: 3 [ 0.047006] ... event mask: 000000070000000f [ 0.048271] rcu: Hierarchical SRCU implementation. [ 0.050237] smp: Bringing up secondary CPUs ... [ 0.051464] x86: Booting SMP configuration: [ 0.052018] .... node #0, CPUs: #1 #2 #3 [ 0.055009] smp: Brought up 1 node, 4 CPUs [ 0.057007] smpboot: Max logical packages: 1 [ 0.058011] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.115086] node 0 deferred pages initialised in 54ms [ 0.119092] devtmpfs: initialized [ 0.121259] x86/mm: Memory block size: 128MB [ 0.124761] gcov: version magic: 0x41383552 [ 0.127241] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.128054] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.129248] pinctrl core: initialized pinctrl subsystem [ 0.130132] [ 0.130553] ************************************************************* [ 0.131010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.132007] ** ** [ 0.133007] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.134008] ** ** [ 0.135007] ** This means that this kernel is built to expose internal ** [ 0.136009] ** IOMMU data structures, which may compromise security on ** [ 0.137008] ** your system. ** [ 0.138008] ** ** [ 0.139008] ** If you see this message and you are not debugging the ** [ 0.140008] ** kernel, report this immediately to your vendor! ** [ 0.141008] ** ** [ 0.142009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.143008] ************************************************************* [ 0.144737] NET: Registered protocol family 16 [ 0.146360] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.148048] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.150043] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.152381] cpuidle: using governor menu [ 0.154929] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.157499] PCI: Using configuration type 1 for base access [ 0.159114] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.169108] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.171030] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.176037] cryptd: max_cpu_qlen set to 1000 [ 0.178214] ACPI: Added _OSI(Module Device) [ 0.180010] ACPI: Added _OSI(Processor Device) [ 0.181008] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.183013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.187830] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.194407] ACPI: Interpreter enabled [ 0.196048] ACPI: PM: (supports S0 S3 S4 S5) [ 0.197010] ACPI: Using IOAPIC for interrupt routing [ 0.199081] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.202355] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.212316] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.215029] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.217012] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.220081] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.226370] acpiphp: Slot [2] registered [ 0.228076] acpiphp: Slot [5] registered [ 0.229086] acpiphp: Slot [6] registered [ 0.231078] acpiphp: Slot [7] registered [ 0.232093] acpiphp: Slot [8] registered [ 0.233076] acpiphp: Slot [9] registered [ 0.235077] acpiphp: Slot [10] registered [ 0.236077] acpiphp: Slot [3] registered [ 0.238071] acpiphp: Slot [4] registered [ 0.240054] acpiphp: Slot [11] registered [ 0.241061] acpiphp: Slot [12] registered [ 0.242057] acpiphp: Slot [13] registered [ 0.243058] acpiphp: Slot [14] registered [ 0.245085] acpiphp: Slot [15] registered [ 0.246073] acpiphp: Slot [16] registered [ 0.248069] acpiphp: Slot [17] registered [ 0.249068] acpiphp: Slot [18] registered [ 0.251070] acpiphp: Slot [19] registered [ 0.252068] acpiphp: Slot [20] registered [ 0.254100] acpiphp: Slot [21] registered [ 0.255071] acpiphp: Slot [22] registered [ 0.257057] acpiphp: Slot [23] registered [ 0.258064] acpiphp: Slot [24] registered [ 0.259071] acpiphp: Slot [25] registered [ 0.261069] acpiphp: Slot [26] registered [ 0.262063] acpiphp: Slot [27] registered [ 0.264105] acpiphp: Slot [28] registered [ 0.265060] acpiphp: Slot [29] registered [ 0.267063] acpiphp: Slot [30] registered [ 0.268065] acpiphp: Slot [31] registered [ 0.270055] PCI host bridge to bus 0000:00 [ 0.271012] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.274014] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.276016] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.278015] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.281013] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.283013] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.285191] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.287943] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.290112] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.299715] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.306053] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.309012] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.311011] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.313011] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.316259] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.318833] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.321041] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.323774] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.329011] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.341013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.345925] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.353736] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.387012] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.414014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.494012] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.509857] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.542011] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.574012] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.664013] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.680000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.693012] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.707013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.740011] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.757616] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.768022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.781018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.832020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.850363] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.863015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.878020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.945014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.965655] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.995018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.028019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.060015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.069300] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.073361] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.075318] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.077339] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.079176] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.085329] iommu: Default domain type: Passthrough [ 1.086375] SCSI subsystem initialized [ 1.087120] ACPI: bus type USB registered [ 1.088076] usbcore: registered new interface driver usbfs [ 1.089066] usbcore: registered new interface driver hub [ 1.090063] usbcore: registered new device driver usb [ 1.092223] pps_core: LinuxPPS API ver. 1 registered [ 1.093007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.096050] PTP clock support registered [ 1.099039] EDAC MC: Ver: 3.0.0 [ 1.101114] PCI: Using ACPI for IRQ routing [ 1.102833] NetLabel: Initializing [ 1.104008] NetLabel: domain hash size = 128 [ 1.105006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.107065] NetLabel: unlabeled traffic allowed by default [ 1.110077] vgaarb: loaded [ 1.112226] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.114011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.119012] clocksource: Switched to clocksource kvm-clock [ 1.221392] VFS: Disk quotas dquot_6.6.0 [ 1.222369] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.225093] *** VALIDATE ramfs *** [ 1.226476] *** VALIDATE hugetlbfs *** [ 1.227661] pnp: PnP ACPI init [ 1.229550] pnp: PnP ACPI: found 6 devices [ 1.297784] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.300633] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.302726] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.304626] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.306314] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.308287] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.310378] NET: Registered protocol family 2 [ 1.312578] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.316562] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.319401] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.323782] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.326683] TCP: Hash tables configured (established 65536 bind 65536) [ 1.329460] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.331714] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.333713] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.336348] NET: Registered protocol family 1 [ 1.339283] RPC: Registered named UNIX socket transport module. [ 1.340814] RPC: Registered udp transport module. [ 1.342495] RPC: Registered tcp transport module. [ 1.344169] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.346474] NET: Registered protocol family 44 [ 1.347698] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.349230] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.350835] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.352965] PCI: CLS 0 bytes, default 64 [ 1.355182] Unpacking initramfs... [ 2.725626] debug: unmapping init [mem 0xffff9817bcc54000-0xffff9817bffbffff] [ 2.729967] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.732510] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.735378] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 3.239591] Initialise system trusted keyrings [ 3.241369] Key type blacklist registered [ 3.243584] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.254254] zbud: loaded [ 3.257741] *** VALIDATE nfs *** [ 3.258951] *** VALIDATE nfs4 *** [ 3.260540] pstore: using deflate compression [ 3.265626] Platform Keyring initialized [ 3.383516] NET: Registered protocol family 38 [ 3.384870] Key type asymmetric registered [ 3.386096] Asymmetric key parser 'x509' registered [ 3.387619] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.390768] io scheduler mq-deadline registered [ 3.392306] io scheduler kyber registered [ 3.393945] io scheduler bfq registered [ 3.396605] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.399919] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.402160] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.404244] ACPI: Power Button [PWRF] [ 3.409203] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.416077] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.474091] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.480691] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.492381] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.520265] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.549233] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.555886] Non-volatile memory driver v1.3 [ 3.557765] Linux agpgart interface v0.103 [ 3.596337] virtio_blk virtio1: [vda] 145880 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.598982] vda: detected capacity change from 0 to 74690560 [ 3.627829] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.630710] vdb: detected capacity change from 0 to 1073741824 [ 3.665282] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.669093] vdc: detected capacity change from 0 to 2621440000 [ 3.693667] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.695928] vdd: detected capacity change from 0 to 2621440000 [ 3.708027] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.710108] vde: detected capacity change from 0 to 4294967296 [ 3.729333] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.731907] vdf: detected capacity change from 0 to 4294967296 [ 3.741204] libphy: Fixed MDIO Bus: probed [ 3.759385] usbcore: registered new interface driver usbserial_generic [ 3.761332] usbserial: USB Serial support registered for generic [ 3.763342] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.767566] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.768990] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.770666] mousedev: PS/2 mouse device common for all mice [ 3.777174] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.777775] rtc_cmos 00:05: RTC can wake from S4 [ 3.781391] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.783463] rtc_cmos 00:05: registered as rtc0 [ 3.787891] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.790050] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.790084] intel_pstate: CPU model not supported [ 3.792231] hid: raw HID events driver (C) Jiri Kosina [ 3.799645] usbcore: registered new interface driver usbhid [ 3.800882] usbhid: USB HID core driver [ 3.801991] drop_monitor: Initializing network drop monitor service [ 3.803700] Initializing XFRM netlink socket [ 3.805138] NET: Registered protocol family 10 [ 3.807994] Segment Routing with IPv6 [ 3.809473] NET: Registered protocol family 17 [ 3.811629] mpls_gso: MPLS GSO support [ 3.817881] RAS: Correctable Errors collector initialized. [ 3.820693] AVX version of gcm_enc/dec engaged. [ 3.822675] AES CTR mode by8 optimization enabled [ 3.891891] sched_clock: Marking stable (3891875272, 0)->(4731072485, -839197213) [ 3.895242] registered taskstats version 1 [ 3.897081] Loading compiled-in X.509 certificates [ 3.898383] zswap: loaded using pool lzo/zbud [ 3.922751] Key type big_key registered [ 3.934447] Key type encrypted registered [ 3.936229] ima: No TPM chip found, activating TPM-bypass! [ 3.938259] ima: Allocated hash algorithm: sha1 [ 3.940238] ima: No architecture policies found [ 3.941829] evm: Initialising EVM extended attributes: [ 3.943927] evm: security.selinux [ 3.945398] evm: security.ima [ 3.946587] evm: security.capability [ 3.947980] evm: HMAC attrs: 0x1 [ 3.950482] rtc_cmos 00:05: setting system clock to 2026-08-03 17:08:50 UTC (1785776930) [ 3.956792] debug: unmapping init [mem 0xffffffff9d003000-0xffffffff9d1fffff] [ 3.962650] debug: unmapping init [mem 0xffffffff9bd82000-0xffffffff9c058fff] [ 3.973091] Write protecting the kernel read-only data: 28672k [ 3.977397] debug: unmapping init [mem 0xffffffff9a403000-0xffffffff9a5fffff] [ 3.981499] debug: unmapping init [mem 0xffffffff9ad14000-0xffffffff9adfffff] [ 4.013578] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.026195] systemd[1]: Detected virtualization kvm. [ 4.029240] systemd[1]: Detected architecture x86-64. [ 4.032564] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.060680] systemd[1]: No hostname configured. [ 4.063257] systemd[1]: Set hostname to . [ 4.066579] random: systemd: uninitialized urandom read (16 bytes read) [ 4.070558] systemd[1]: Initializing machine ID from random generator. [ 4.137432] random: ln: uninitialized urandom read (6 bytes read) [ 4.217202] random: systemd: uninitialized urandom read (16 bytes read) [ 4.219915] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.223129] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 4.226871] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.792918] device-mapper: uevent: version 1.0.3 [ 4.796092] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.553836] virtio_net virtio0 ens2: renamed from eth0 [ 5.591194] random: fast init done [ 5.650703] scsi host0: ata_piix [ 5.706279] scsi host1: ata_piix [ 5.708249] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.710872] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.153925] dracut-initqueue[586]: RTNETLINK answers: File exists [ 10.456295] random: crng init done [ 10.457815] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.968862] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.119464] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.400951] SELinux: Disabled at runtime. [ 12.464921] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.472200] systemd[1]: Detected virtualization kvm. [ 12.473749] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.944456] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.947540] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.951612] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.954716] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.958166] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.965119] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.971971] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on[ 13.094063] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Process Core Dump Socket. Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.434125] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.673565] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.735975] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.824915] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.843559] EDAC sbridge: Ver: 1.1.2 [ 18.669191] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (5s / no limit) [** ] A start job is running for Configur…-only root support (6s / no limit)[ 19.545951] NFS: Registering the id_resolver key type [ 19.551783] Key type id_resolver registered [ 19.558299] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg407-server login: [ 35.693759] hrtimer: interrupt took 3751955 ns [ 113.548665] libcfs: loading out-of-tree module taints kernel. [ 113.636623] Key type ._llcrypt registered [ 113.639274] Key type .llcrypt registered [ 113.718424] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 140.163732] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 142.278027] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 142.291553] alg: No test for adler32 (adler32-zlib) [ 144.162924] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [ 145.558625] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 147.535209] Key type lgssc registered [ 150.408383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 171.208992] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 229.312987] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 248.022838] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 248.079674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 249.465187] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 249.495286] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 249.632916] Lustre: lustre-MDT0000: new disk, initializing [ 249.760275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 249.798480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 256.146607] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 274.127094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 274.257323] Lustre: 6543:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 274.309727] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 274.316162] Lustre: Skipped 1 previous similar message [ 274.432098] Lustre: lustre-MDT0001: new disk, initializing [ 274.551636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 274.585636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 274.609446] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 280.181447] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 286.128947] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 298.606743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 299.267168] Lustre: lustre-OST0000: new disk, initializing [ 299.274759] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 299.282748] Lustre: 8482:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 299.389458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 307.234344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 307.251609] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 307.353586] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 308.024065] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 326.865585] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 327.051155] Lustre: lustre-OST0001: new disk, initializing [ 327.057899] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 327.065493] Lustre: 9555:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 327.128827] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 336.533961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 336.578477] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 336.755905] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 337.477086] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 355.638570] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 363.579195] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 374.017946] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing check_logdir /tmp/testlogs/ [ 380.508660] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing yml_node [ 386.105199] Lustre: DEBUG MARKER: Client: 2.17.56.2 [ 390.065310] Lustre: DEBUG MARKER: MDS: 2.17.56.2 [ 394.100504] Lustre: DEBUG MARKER: OSS: 2.17.56.2 [ 396.890226] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 3 13:15:20 EDT 2026 [ 424.327995] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 427.236056] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 448.480050] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 448.488778] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 448.499699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 449.504537] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 449.506264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 449.519074] Lustre: Skipped 1 previous similar message [ 449.530762] Lustre: Skipped 1 previous similar message [ 454.631840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 454.653279] Lustre: Skipped 3 previous similar messages [ 455.109850] Lustre: server umount lustre-MDT0000 complete [ 461.559877] LustreError: 6534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785777388 with bad export cookie 17229379449403933587 [ 461.579060] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 461.590030] LustreError: 6534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 464.864969] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 464.871323] LustreError: 6551:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 464.887740] Lustre: Skipped 1 previous similar message [ 464.896233] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 464.922168] LustreError: 6551:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 464.973551] Lustre: Skipped 1 previous similar message [ 466.912056] LustreError: 12502:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 466.941113] LustreError: 12502:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 468.503149] Lustre: server umount lustre-MDT0001 complete [ 474.218535] Lustre: server umount lustre-OST0000 complete [ 479.629407] Lustre: server umount lustre-OST0001 complete [ 490.155193] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 502.314339] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 553.131529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 553.451252] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 553.485768] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 553.592592] Lustre: lustre-MDT0000: new disk, initializing [ 553.692910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 553.729726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 561.336176] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 575.788547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 575.989301] Lustre: 15114:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 576.073420] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 576.079621] Lustre: Skipped 1 previous similar message [ 576.242934] Lustre: lustre-MDT0001: new disk, initializing [ 576.426552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 576.479425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 576.494128] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 585.175584] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 601.231835] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 612.037882] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 622.373415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 622.899290] Lustre: lustre-OST0000: new disk, initializing [ 622.906410] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 622.916223] Lustre: 17087:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 623.124875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 624.940057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 624.952264] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 625.106858] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 629.968687] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 642.998426] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 651.250790] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 651.417863] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 659.058793] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 659.278470] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 666.084811] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 666.086134] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 666.090288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 666.090300] Lustre: Skipped 2 previous similar messages [ 666.108446] LustreError: Skipped 1 previous similar message [ 666.160132] Lustre: Skipped 2 previous similar messages [ 667.868959] Lustre: server umount lustre-OST0000 complete [ 673.761677] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 673.780147] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 676.325139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 676.343581] Lustre: Skipped 3 previous similar messages [ 679.997937] Lustre: server umount lustre-MDT0000 complete [ 681.440652] LustreError: 15126:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 681.485753] LustreError: 15126:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 685.536183] LustreError: 15127:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 686.172091] LustreError: 15108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785777612 with bad export cookie 17229379449403935120 [ 686.185165] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 686.187298] LustreError: 15108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 686.918418] Lustre: server umount lustre-MDT0001 complete [ 692.467965] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 13:20:16 (1785777616) [ 695.417230] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 698.296237] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 13:20:22 (1785777622) [ 709.973911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 710.933629] LustreError: 19499:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 710.965275] LustreError: 19499:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 711.101556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 718.290658] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 721.376378] LustreError: 19500:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 721.395477] LustreError: 19500:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 731.713650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 738.114581] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 748.513416] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 756.536682] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 765.254232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 774.056428] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 785.381345] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 796.659105] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 796.672953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 796.688777] Lustre: Skipped 2 previous similar messages [ 796.689115] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 796.732594] Lustre: Skipped 1 previous similar message [ 802.295372] Lustre: server umount lustre-OST0000 complete [ 808.929020] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 813.585368] Lustre: server umount lustre-MDT0000 complete [ 817.123718] LustreError: 19499:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 817.150446] LustreError: 19499:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 818.059933] LustreError: 20235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785777744 with bad export cookie 17229379449403936002 [ 818.070394] LustreError: 20235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 818.072499] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 818.843152] Lustre: server umount lustre-MDT0001 complete [ 837.660152] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 842.600020] Key type lgssc unregistered [ 843.082602] LNet: 23054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 843.103549] LNetError: 23054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 843.170603] LNet: Removed LNI 192.168.204.107@tcp [ 845.027255] Key type .llcrypt unregistered [ 845.034199] Key type ._llcrypt unregistered [ 875.861307] Key type ._llcrypt registered [ 875.864798] Key type .llcrypt registered [ 876.048528] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 877.905663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 878.172202] alg: No test for adler32 (adler32-zlib) [ 879.957031] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [ 880.655495] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 882.495211] Key type lgssc registered [ 885.018351] Lustre: Echo OBD driver; http://www.lustre.org/ [ 898.879446] loop: module loaded [ 899.004337] loop0: detected capacity change from 0 to 204800000 [ 899.425854] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 899.455926] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 900.252857] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 910.095577] loop0: detected capacity change from 0 to 204800000 [ 910.691271] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 910.727611] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 910.757028] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 920.065925] loop0: detected capacity change from 0 to 204800000 [ 927.440136] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 927.467854] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 928.812328] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 928.914798] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 929.050729] Lustre: lustre-MDT0000: new disk, initializing [ 929.243260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 929.295972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 935.977787] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 952.174390] loop1: detected capacity change from 0 to 204800000 [ 961.578713] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 961.976833] Lustre: lustre-OST07c6: new disk, initializing [ 961.983805] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 961.994991] Lustre: Skipped 1 previous similar message [ 962.008418] Lustre: 26255:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 962.112074] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 971.386224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 971.406923] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 971.456395] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 971.463740] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 978.921860] Lustre: 25242:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 978.943418] Lustre: 25242:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 978.952566] Lustre: 25242:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 978.974266] Lustre: 25242:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 978.986980] Lustre: 25242:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 979.003299] Lustre: 25242:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 997.578762] Lustre: 25241:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 997.590774] Lustre: 25241:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 997.599307] Lustre: 25241:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 997.610912] Lustre: 25241:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 997.624413] Lustre: 25241:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 997.634074] Lustre: 25241:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1003.638945] Lustre: 25241:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 1003.654978] Lustre: 25241:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 1003.668156] Lustre: 25241:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 1003.679618] Lustre: 25241:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1003.696511] Lustre: 25241:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1003.717100] Lustre: 25241:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1003.735274] Lustre: 25241:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 1003.754405] Lustre: 25241:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1003.771297] Lustre: 25241:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 1003.784364] Lustre: 25241:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1003.802096] Lustre: 25241:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1003.819874] Lustre: 25241:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1007.226688] Lustre: 25242:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 1007.241322] Lustre: 25242:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 1007.259864] Lustre: 25242:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 1007.268737] Lustre: 25242:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1007.291850] Lustre: 25242:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1007.307461] Lustre: 25242:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1007.324915] Lustre: 25242:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 1007.332993] Lustre: 25242:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1007.337231] Lustre: 25242:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 1007.342263] Lustre: 25242:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1007.351047] Lustre: 25242:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1007.357285] Lustre: 25242:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1017.331403] Lustre: 25241:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 1017.356901] Lustre: 25241:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 1017.373018] Lustre: 25241:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 1017.396917] Lustre: 25241:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 1017.405250] Lustre: 25241:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1017.415481] Lustre: 25241:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 1017.421878] Lustre: 25241:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 1017.429774] Lustre: 25241:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 1017.434225] Lustre: 25241:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 1017.441546] Lustre: 25241:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 1017.445570] Lustre: 25241:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1017.448416] Lustre: 25241:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 1032.074985] Lustre: 25243:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 1032.108893] Lustre: 25243:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 1032.131120] Lustre: 25243:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 1032.152136] Lustre: 25243:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 1032.181939] Lustre: 25243:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1032.207617] Lustre: 25243:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 1032.228770] Lustre: 25243:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 1032.264122] Lustre: 25243:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 1032.282686] Lustre: 25243:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 1032.313633] Lustre: 25243:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 1032.338077] Lustre: 25243:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1032.362866] Lustre: 25243:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 1088.992542] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1089.003836] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1089.047287] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 1090.655879] Lustre: server umount lustre-OST07c6 complete [ 1102.894557] Lustre: server umount lustre-MDT0000 complete [ 1125.883324] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 1131.391526] Key type lgssc unregistered [ 1132.046663] LNet: 28818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1132.059579] LNetError: 28818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1132.109759] LNet: Removed LNI 192.168.204.107@tcp [ 1133.344125] Key type .llcrypt unregistered [ 1133.345555] Key type ._llcrypt unregistered [ 1160.938559] Key type ._llcrypt registered [ 1160.942380] Key type .llcrypt registered [ 1161.072723] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 1162.487964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1162.705359] alg: No test for adler32 (adler32-zlib) [ 1163.922183] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [ 1164.266987] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 1166.064266] Key type lgssc registered [ 1167.061766] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1175.034776] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1175.055960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1176.597354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1181.725864] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1186.787458] LustreError: 30094:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1186.818239] LustreError: 30094:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1191.907920] LustreError: 30095:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1192.048777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1192.566583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1197.542078] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1208.449474] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1217.899423] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1228.963578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1229.612717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1239.504988] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1247.112625] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1251.727559] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1277.301521] Lustre: server umount lustre-OST0000 complete [ 1284.578398] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1284.620240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1284.649707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1288.166312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1288.171557] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1293.295526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1293.309073] Lustre: Skipped 2 previous similar messages [ 1294.367530] Lustre: 29405:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785778204/real 1785778204] req@000000008c438540 x1872524046536960/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1785778220 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1294.399546] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1294.428926] LustreError: 30094:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1294.453638] LustreError: 30094:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1294.633233] Lustre: server umount lustre-MDT0000 complete [ 1299.429828] Lustre: 29402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785778209/real 1785778209] req@000000007f61efea x1872524046538752/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785778225 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1299.936800] LustreError: 30089:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1299.962667] LustreError: 30089:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1300.800677] LustreError: 30075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785778227 with bad export cookie 7415139729310122216 [ 1300.810919] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1300.816251] LustreError: 30075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1301.499764] Lustre: server umount lustre-MDT0001 complete [ 1318.070845] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 1322.893484] Key type lgssc unregistered [ 1323.579175] LNet: 33836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1323.592772] LNetError: 33836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1323.624515] LNet: Removed LNI 192.168.204.107@tcp [ 1325.509763] Key type .llcrypt unregistered [ 1325.519938] Key type ._llcrypt unregistered [ 1359.533718] Key type ._llcrypt registered [ 1359.541183] Key type .llcrypt registered [ 1359.729229] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 1362.041509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1362.128969] alg: No test for adler32 (adler32-zlib) [ 1363.508493] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [ 1364.166564] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 1365.935358] Key type lgssc registered [ 1368.042810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1380.698616] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1380.717174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1382.531670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1390.302062] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1392.101041] LustreError: 35109:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1392.128713] LustreError: 35109:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1397.222230] LustreError: 35110:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1402.346401] LustreError: 35109:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1405.210048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1406.411392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1412.656520] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1424.340308] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1433.589446] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1443.917285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1444.463483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1451.665883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1453.600649] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1464.770694] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1481.696760] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1481.708624] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1481.722822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1482.724686] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1482.736424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1482.740495] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1482.776218] Lustre: Skipped 1 previous similar message [ 1485.330540] Lustre: server umount lustre-OST0000 complete [ 1492.966037] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1492.987192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1493.003493] Lustre: Skipped 1 previous similar message [ 1493.028453] Lustre: Skipped 1 previous similar message [ 1497.501324] Lustre: server umount lustre-MDT0000 complete [ 1498.082881] LustreError: 35109:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1498.095593] LustreError: 35109:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1502.016625] LustreError: 35091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785778428 with bad export cookie 16492589363362818613 [ 1502.028780] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1502.045751] LustreError: 35091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1503.029487] Lustre: server umount lustre-MDT0001 complete [ 1519.039637] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 1523.858538] Key type lgssc unregistered [ 1524.219573] LNet: 38673:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1524.230119] LNetError: 38673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1524.271320] LNet: Removed LNI 192.168.204.107@tcp [ 1525.318390] Key type .llcrypt unregistered [ 1525.322714] Key type ._llcrypt unregistered [ 1560.843944] Key type ._llcrypt registered [ 1560.856590] Key type .llcrypt registered [ 1561.196984] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 1565.134408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1565.168753] alg: No test for adler32 (adler32-zlib) [ 1566.604316] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [ 1567.092609] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 1568.968037] Key type lgssc registered [ 1571.186404] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1583.556797] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1583.614233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1585.642857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1590.578263] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1595.884140] LustreError: 40093:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1595.916376] LustreError: 40093:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1601.004801] LustreError: 40094:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1601.934131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1602.498272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1608.357543] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1619.500768] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1626.541695] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1637.153551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1637.788239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1646.650353] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1656.277731] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1659.159599] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1669.091302] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1669.104250] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1669.131107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1671.661369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1671.661469] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1674.965028] Lustre: server umount lustre-MDT0000 complete [ 1676.775748] LustreError: 40093:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1676.838219] LustreError: 40093:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1681.052401] LustreError: 40829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785778607 with bad export cookie 10428175920629212539 [ 1681.059805] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1681.071376] LustreError: 40829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1681.899268] LustreError: 40090:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1681.899854] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1681.927742] LustreError: 40090:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1681.976940] Lustre: Skipped 1 previous similar message [ 1681.983852] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1682.003729] Lustre: Skipped 3 previous similar messages [ 1687.013034] LustreError: 40089:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1687.014443] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1687.065618] LustreError: 40089:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1687.077508] Lustre: Skipped 1 previous similar message [ 1688.312627] Lustre: server umount lustre-MDT0001 complete [ 1707.770928] Lustre: server umount lustre-OST0000 complete [ 1731.647907] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 1735.637532] Key type lgssc unregistered [ 1736.013802] LNet: 43962:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1736.029917] LNetError: 43962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1736.049250] LNet: Removed LNI 192.168.204.107@tcp [ 1737.147968] Key type .llcrypt unregistered [ 1737.151510] Key type ._llcrypt unregistered [ 1773.681438] Key type ._llcrypt registered [ 1773.687296] Key type .llcrypt registered [ 1773.894296] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 1775.463582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1775.536098] alg: No test for adler32 (adler32-zlib) [ 1776.803660] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [ 1777.075243] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 1778.751161] Key type lgssc registered [ 1780.639035] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1791.085987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1791.127728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1792.836574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1797.931241] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1803.236825] LustreError: 45386:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1803.262310] LustreError: 45386:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1808.356204] LustreError: 45385:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1810.743493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1811.347542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1817.909623] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1831.247532] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1839.678107] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1849.413259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1850.182746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1858.973711] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1862.517548] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1871.459849] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1882.092532] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1882.111880] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1882.149393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1883.115048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1883.116680] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1883.127096] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1885.909850] Lustre: server umount lustre-OST0000 complete [ 1907.681299] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1907.684397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1907.695996] Lustre: Skipped 1 previous similar message [ 1907.711624] Lustre: Skipped 3 previous similar messages [ 1913.687428] Lustre: server umount lustre-MDT0000 complete [ 1913.831150] LustreError: 45386:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1913.860960] LustreError: 45386:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1918.760927] LustreError: 45366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785778845 with bad export cookie 5967115303844799403 [ 1918.781671] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1933.279188] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1933.831836] Lustre: server umount lustre-MDT0001 complete [ 1948.937754] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 1952.868517] Key type lgssc unregistered [ 1953.338592] LNet: 49102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1953.359948] LNetError: 49102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1954.406455] LNet: Removed LNI 192.168.204.107@tcp [ 1955.851670] Key type .llcrypt unregistered [ 1955.866824] Key type ._llcrypt unregistered [ 1991.416804] Key type ._llcrypt registered [ 1991.423221] Key type .llcrypt registered [ 1991.589893] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 1992.974241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1993.119672] alg: No test for adler32 (adler32-zlib) [ 1994.604624] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [ 1994.965596] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 1996.711163] Key type lgssc registered [ 1997.820566] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2007.347179] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2007.356338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2008.955985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2015.239347] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2019.297155] LustreError: 50520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2019.333379] LustreError: 50520:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2024.418270] LustreError: 50521:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2029.425140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2029.538908] LustreError: 50520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2030.297740] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2040.578580] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2051.069655] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2058.381781] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2068.229588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2069.142058] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2079.122890] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2079.477119] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 2089.929650] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2096.455159] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 2101.874385] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 2115.252789] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-0313h43m57s [ 2117.709116] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 2122.211339] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2122.222693] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2122.248831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2122.995177] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.7@tcp (stopping) [ 2123.005705] Lustre: Skipped 3 previous similar messages [ 2124.044234] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.7@tcp (stopping) [ 2124.053409] Lustre: Skipped 3 previous similar messages [ 2127.604897] Lustre: server umount lustre-MDT0000 complete [ 2129.961951] LustreError: 50515:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2130.009147] LustreError: 50515:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2132.273197] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 2136.053457] LustreError: 53409:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2136.078143] LustreError: 53409:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2140.886892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2141.037314] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2141.441510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2141.537274] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 2142.454366] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2148.788186] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2153.251184] Lustre: DEBUG MARKER: Wait for df (36731) ... [ 2156.135970] Lustre: DEBUG MARKER: done [ 2167.264832] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2167.281060] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2167.311507] Lustre: Skipped 2 previous similar messages [ 2167.319738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2171.390608] Lustre: server umount lustre-OST0000 complete [ 2178.528615] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2178.531524] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2178.551470] LustreError: Skipped 1 previous similar message [ 2178.582542] Lustre: Skipped 1 previous similar message [ 2178.586970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2178.586996] Lustre: Skipped 3 previous similar messages [ 2183.001248] Lustre: server umount lustre-MDT0000 complete [ 2186.221932] LustreError: 53016:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2186.248178] LustreError: 53016:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2189.104329] LustreError: 50500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785779115 with bad export cookie 5771074334406888424 [ 2189.111903] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2189.127038] LustreError: 50500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2190.202735] Lustre: server umount lustre-MDT0001 complete [ 2207.494997] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 2212.092887] Key type lgssc unregistered [ 2212.820455] LNet: 55453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2212.835908] LNetError: 55453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2212.873125] LNet: Removed LNI 192.168.204.107@tcp [ 2215.259544] Key type .llcrypt unregistered [ 2215.265806] Key type ._llcrypt unregistered [ 2284.506551] Key type ._llcrypt registered [ 2284.510468] Key type .llcrypt registered [ 2284.695549] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 2287.049291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2287.199719] alg: No test for adler32 (adler32-zlib) [ 2288.819507] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [ 2289.387451] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 2291.199333] Key type lgssc registered [ 2293.744462] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2310.732698] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2310.767790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2312.202113] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2312.218819] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2312.457219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2318.621564] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2331.212763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2331.350225] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2331.416210] Lustre: 57669:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2331.793727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2338.298626] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2351.609537] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2362.281684] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2373.290906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2373.605924] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2373.829961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2380.514136] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 2384.183987] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2395.172560] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2403.329381] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-0313h48m46s [ 2405.833499] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 2415.786353] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 2419.455660] Lustre: DEBUG MARKER: Creating a test file and stat it [ 2419.512256] LustreError: 60253:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 2439.980780] Lustre: lustre-MDT0000: Client c773171f-b573-4891-a175-7c0af03d770f (at 192.168.204.7@tcp) reconnecting [ 2459.599731] LustreError: 60253:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 2462.833459] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2466.997666] Lustre: DEBUG MARKER: done [ 2478.067843] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2478.089104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2478.103144] Lustre: Skipped 1 previous similar message [ 2478.111118] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2478.113284] Lustre: Skipped 1 previous similar message [ 2483.196509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2483.218448] Lustre: Skipped 1 previous similar message [ 2484.300243] Lustre: server umount lustre-OST0000 complete [ 2490.339293] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2490.349809] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2490.378018] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2494.518152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2494.538756] Lustre: Skipped 4 previous similar messages [ 2496.947228] Lustre: server umount lustre-MDT0000 complete [ 2502.106316] LustreError: 57661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785779428 with bad export cookie 13284206875932558444 [ 2502.109314] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2502.120940] LustreError: 57661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2502.830316] Lustre: server umount lustre-MDT0001 complete [ 2517.835627] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 2522.082888] Key type lgssc unregistered [ 2522.576933] LNet: 62007:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2522.596415] LNetError: 62007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2522.627546] LNet: Removed LNI 192.168.204.107@tcp [ 2523.998886] Key type .llcrypt unregistered [ 2524.002784] Key type ._llcrypt unregistered [ 2591.225390] Key type ._llcrypt registered [ 2591.227187] Key type .llcrypt registered [ 2591.403906] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 2593.761152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2593.814521] alg: No test for adler32 (adler32-zlib) [ 2595.079081] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [ 2595.341294] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 2597.063165] Key type lgssc registered [ 2598.657467] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2603.623706] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2603.654381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2605.190210] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2605.219344] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2605.394434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2606.720996] Lustre: Failing over lustre-MDT0000 [ 2607.292677] Lustre: server umount lustre-MDT0000 complete [ 2616.118267] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 13:52:20 (1785779540) [ 2623.495376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2623.970689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2629.479927] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2642.280659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2642.423613] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2642.493747] Lustre: 64828:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2643.001653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2649.807288] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2660.369206] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2669.279882] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2676.754800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2677.070637] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2677.409846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2683.134900] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2685.945415] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2695.191387] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2702.221716] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2714.085042] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2714.086636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2714.114600] Lustre: Skipped 2 previous similar messages [ 2714.117336] Lustre: Skipped 1 previous similar message [ 2718.142082] Lustre: server umount lustre-MDT0000 complete [ 2723.528693] LustreError: 64820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785779650 with bad export cookie 1680086710557229630 [ 2723.536443] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2723.544183] LustreError: 64820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2724.185545] Lustre: server umount lustre-MDT0001 complete [ 2728.500160] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2736.892454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2739.679249] Lustre: 63749:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785779650/real 1785779650] req@000000003f13305c x1872525547056128/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785779666 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2739.721625] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2743.779596] Lustre: 63752:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785779654/real 1785779654] req@00000000f2563a3c x1872525547056256/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785779670 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2744.863086] Lustre: 63752:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785779655/real 1785779655] req@000000008513d57a x1872525547056384/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785779671 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2748.467067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2749.450683] LustreError: 66676:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2749.471659] LustreError: 66676:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2749.580723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2749.705157] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2750.559308] Lustre: 63749:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785779660/real 1785779660] req@00000000b8966529 x1872525547056640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785779676 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2750.625109] LustreError: 68601:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2750.648263] LustreError: 68601:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2754.726843] Lustre: 63750:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785779665/real 1785779665] req@000000009772dc45 x1872525547056768/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785779681 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2755.305429] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2756.066253] LustreError: 68600:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2756.091784] LustreError: 68600:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2761.195239] LustreError: 66677:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2761.212797] LustreError: 66677:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2766.307830] LustreError: 68665:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2766.460125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2766.962076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2769.005034] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2773.531465] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2782.869587] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2791.024056] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2802.675623] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2802.688115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2802.712247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2802.722831] Lustre: Skipped 1 previous similar message [ 2807.497220] Lustre: server umount lustre-MDT0000 complete [ 2808.298776] LustreError: 68665:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2808.330444] LustreError: 68665:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2812.193558] LustreError: 64820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785779738 with bad export cookie 1680086710557231905 [ 2812.206612] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2812.213614] LustreError: 64820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2812.798407] Lustre: server umount lustre-MDT0001 complete [ 2821.382720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2828.769821] Lustre: 63749:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785779739/real 1785779739] req@0000000008756de9 x1872525547087360/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785779755 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2828.806546] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2828.823490] Lustre: Skipped 2 previous similar messages [ 2833.684130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2837.088338] LustreError: 63748:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000093f9b5ae x1872525547088384/t0(0) o250->MGC192.168.204.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2837.763876] LustreError: 66677:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2837.902606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2838.045033] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2844.601743] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2857.843528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2859.529033] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2859.538795] Lustre: Skipped 1 previous similar message [ 2864.195189] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2873.525271] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2880.907678] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2894.304286] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2894.311891] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2894.342553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2894.346027] Lustre: Skipped 3 previous similar messages [ 2897.013378] Lustre: server umount lustre-MDT0000 complete [ 2900.453660] LustreError: 69414:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2900.479690] LustreError: 69414:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 2901.886376] LustreError: 64820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785779828 with bad export cookie 1680086710557233284 [ 2901.887298] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2901.901295] LustreError: 64820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2902.571863] Lustre: server umount lustre-MDT0001 complete [ 2910.750635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2915.679037] Lustre: DEBUG MARKER: files compared the same [ 2929.223724] Lustre: server umount lustre-OST0000 complete [ 2954.215778] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 2958.757960] Key type lgssc unregistered [ 2959.304917] LNet: 74567:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2959.341886] LNetError: 74567:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2960.430739] LNet: Removed LNI 192.168.204.107@tcp [ 2962.049190] Key type .llcrypt unregistered [ 2962.054189] Key type ._llcrypt unregistered [ 2992.815227] Key type ._llcrypt registered [ 2992.825572] Key type .llcrypt registered [ 2992.959857] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 2994.533332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2994.821190] alg: No test for adler32 (adler32-zlib) [ 2995.978229] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [ 2996.382555] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 2998.095160] Key type lgssc registered [ 2999.906340] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3013.592404] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3013.617184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3015.434266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3024.104337] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug +malloc all [ 3024.869228] LustreError: 75987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3024.911971] LustreError: 75987:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3030.013985] LustreError: 75988:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3035.106388] LustreError: 75987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3038.349729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3038.799350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3043.721710] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug +malloc all [ 3053.573915] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3062.502304] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3075.526171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3076.628357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3084.863931] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 3088.670729] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug +malloc all [ 3102.872905] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3114.464069] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3114.479362] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3114.513222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3115.507598] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3115.517127] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3115.525397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3120.304755] Lustre: server umount lustre-OST0000 complete [ 3125.728530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3125.730643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3125.746520] Lustre: Skipped 1 previous similar message [ 3125.751030] Lustre: Skipped 1 previous similar message [ 3130.851642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3130.859895] Lustre: Skipped 2 previous similar messages [ 3132.382846] Lustre: server umount lustre-MDT0000 complete [ 3135.973344] LustreError: 75987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3136.013780] LustreError: 75987:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3140.166720] LustreError: 75969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785780066 with bad export cookie 6386681143315981290 [ 3140.178906] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3140.194263] LustreError: 75969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3141.265463] Lustre: server umount lustre-MDT0001 complete [ 3160.008603] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 3163.790878] Key type lgssc unregistered [ 3164.164136] LNet: 79539:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3164.172497] LNetError: 79539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3164.194630] LNet: Removed LNI 192.168.204.107@tcp [ 3165.291991] Key type .llcrypt unregistered [ 3165.296692] Key type ._llcrypt unregistered [ 3205.460099] Key type ._llcrypt registered [ 3205.474593] Key type .llcrypt registered [ 3205.779565] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 3208.607923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3208.817622] alg: No test for adler32 (adler32-zlib) [ 3210.511745] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [ 3211.193082] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 3213.055826] Key type lgssc registered [ 3216.932840] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3228.651914] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3228.684947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3244.383594] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3249.504293] LustreError: 80936:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.107@tcp: failed processing log, type 4: rc = -110 [ 3276.256875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3286.822547] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3297.078466] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3307.970412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3308.165924] Lustre: *** cfs_fail_loc=706, val=0*** [ 3308.904486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3309.043751] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 3315.387497] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3316.193747] LustreError: 82023:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3316.223435] LustreError: 82023:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 3321.333612] LustreError: 82119:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3321.365993] LustreError: 82119:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3326.438127] LustreError: 82024:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3326.466154] LustreError: 82024:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3327.169989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3327.756967] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3333.147967] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3344.218626] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3352.251753] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3355.105565] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3355.122679] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3355.160246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3358.694845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3358.697747] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3358.722263] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3361.153205] Lustre: server umount lustre-OST0000 complete [ 3368.938839] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3368.943971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3368.949742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3368.949751] Lustre: Skipped 1 previous similar message [ 3368.969335] Lustre: Skipped 1 previous similar message [ 3372.806622] Lustre: server umount lustre-MDT0000 complete [ 3374.059161] LustreError: 82027:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3374.093022] LustreError: 82027:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3377.472517] LustreError: 80943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785780304 with bad export cookie 12284856836490358081 [ 3377.493887] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3377.977439] Lustre: server umount lustre-MDT0001 complete [ 3394.014479] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 3397.904129] Key type lgssc unregistered [ 3398.344666] LNet: 84555:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3398.348848] LNetError: 84555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3399.399628] LNet: Removed LNI 192.168.204.107@tcp [ 3400.854492] Key type .llcrypt unregistered [ 3400.858929] Key type ._llcrypt unregistered [ 3437.954866] Key type ._llcrypt registered [ 3437.961859] Key type .llcrypt registered [ 3438.144578] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 3439.478108] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3439.586489] alg: No test for adler32 (adler32-zlib) [ 3440.702303] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [ 3440.896129] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 3442.538383] Key type lgssc registered [ 3444.615126] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3456.678378] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3456.772793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3463.090695] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3475.475802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3476.877995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3482.081736] LustreError: 86678:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3482.123858] LustreError: 86678:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3487.204968] LustreError: 86677:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3487.234721] LustreError: 86677:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3487.472442] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3492.327077] LustreError: 86682:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3492.399541] LustreError: 86682:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3497.449522] LustreError: 86681:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3497.480016] LustreError: 86681:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3501.541888] LustreError: 86678:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3501.565612] LustreError: 86678:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3503.827134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3504.577160] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3509.732842] LustreError: 86676:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3509.780862] LustreError: 86676:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 3514.513476] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3527.141760] LustreError: 86682:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3527.163072] LustreError: 86682:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 3528.318982] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3528.330078] Lustre: MGS: Client e7441960-1bb7-4eae-a8b7-698bbd81c319 (at 0@lo) reconnecting [ 3528.362991] LustreError: 86692:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3528.708223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3528.786918] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 3535.223911] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3554.285050] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3554.291281] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3554.303385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3554.728928] Lustre: server umount lustre-OST0000 complete [ 3564.528424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3564.529236] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3564.533940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3564.533951] Lustre: Skipped 1 previous similar message [ 3564.539969] Lustre: Skipped 2 previous similar messages [ 3566.352567] Lustre: server umount lustre-MDT0000 complete [ 3569.646782] LustreError: 86682:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3569.671040] LustreError: 86682:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3571.681090] Lustre: server umount lustre-MDT0001 complete [ 3576.759631] Lustre: server umount MGS complete [ 3593.356227] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 14:08:37 (1785780517) [ 3631.844708] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 3648.865847] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 3718.418225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3724.373766] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3738.630243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3738.892932] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3738.934805] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3739.027105] Lustre: lustre-MDT0001: new disk, initializing [ 3739.100193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3739.138791] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3739.161436] Lustre: 93901:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3739.179379] LustreError: 93901:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3744.611403] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3758.093264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3758.548544] Lustre: lustre-OST0000: new disk, initializing [ 3758.554117] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3758.563500] Lustre: 94667:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3758.648385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3766.867610] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3768.938543] Lustre: 95047:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000835701cc x1872526434152832/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1785780711 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3778.894492] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3786.954672] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3786.986778] Lustre: MGS: Client 6ee39524-65b0-4779-a7f8-f6d78c47d66a (at 0@lo) reconnecting [ 3787.055688] Lustre: 93168:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3787.074375] LustreError: 93887:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3787.087031] LustreError: 93887:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 3787.124323] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3787.248317] Lustre: lustre-MDT0000: new disk, initializing [ 3787.451421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3787.489157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3787.529456] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3788.598346] Lustre: 95738:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000005528c4cb x1872526434159744/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1785780731 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3792.857692] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3793.792847] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3796.480467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3796.494300] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3797.673911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 3797.692984] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 3797.884674] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 3827.170053] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3827.181872] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3827.210308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3827.218394] Lustre: Skipped 1 previous similar message [ 3830.249097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3830.264926] Lustre: Skipped 2 previous similar messages [ 3831.698094] Lustre: server umount lustre-OST0000 complete [ 3840.482764] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3840.484535] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3840.495905] LustreError: Skipped 1 previous similar message [ 3840.499802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3840.550045] Lustre: Skipped 2 previous similar messages [ 3843.758364] Lustre: server umount lustre-MDT0000 complete [ 3845.608662] LustreError: 93877:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3845.652090] LustreError: 93877:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3857.821330] Lustre: server umount lustre-MDT0001 complete [ 3866.466939] Lustre: server umount MGS complete [ 3887.319943] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 14:13:31 (1785780811) [ 3913.317869] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 3918.463184] Key type lgssc unregistered [ 3919.015976] LNet: 98240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3919.032651] LNetError: 98240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3919.083063] LNet: Removed LNI 192.168.204.107@tcp [ 3921.066250] Key type .llcrypt unregistered [ 3921.069082] Key type ._llcrypt unregistered [ 3946.470288] Key type ._llcrypt registered [ 3946.471839] Key type .llcrypt registered [ 3946.603663] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 3947.981154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3948.212050] alg: No test for adler32 (adler32-zlib) [ 3949.354304] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [ 3949.828000] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 3951.631309] Key type lgssc registered [ 3953.499790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3966.389296] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3966.461371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3968.388464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3976.213081] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3977.703242] LustreError: 99463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3977.745754] LustreError: 99463:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3982.823984] LustreError: 99464:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3987.937077] LustreError: 99463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3988.856304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3989.342899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3997.035549] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4010.070584] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4019.325660] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4032.860228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4033.251670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4038.456031] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 4042.283399] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4055.400923] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4073.207684] Lustre: server umount lustre-OST0000 complete [ 4079.586765] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4079.613445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4079.626189] Lustre: Skipped 1 previous similar message [ 4084.705220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4084.716684] Lustre: Skipped 1 previous similar message [ 4089.827523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4089.845966] Lustre: Skipped 1 previous similar message [ 4090.783273] Lustre: 98776:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785781001/real 1785781001] req@000000003e1066fb x1872526967869824/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785781017 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4090.821477] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4090.833293] Lustre: Skipped 1 previous similar message [ 4091.871272] Lustre: 98776:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785781002/real 1785781002] req@00000000a8f3ff3f x1872526967870336/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1785781018 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4093.409149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4093.674462] Lustre: server umount lustre-MDT0000 complete [ 4095.970529] LustreError: 100925:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4096.018703] LustreError: 100925:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4097.000292] Lustre: 98776:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785781006/real 1785781006] req@00000000b8a06b8d x1872526967871744/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785781022 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4100.557672] LustreError: 99445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785781027 with bad export cookie 11255991852412313506 [ 4100.570275] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4100.574017] LustreError: 99445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4101.378163] Lustre: server umount lustre-MDT0001 complete [ 4126.178643] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 4130.567384] Key type lgssc unregistered [ 4130.999331] LNet: 103044:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4131.010555] LNetError: 103044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4131.027665] LNet: Removed LNI 192.168.204.107@tcp [ 4132.520402] Key type .llcrypt unregistered [ 4132.522079] Key type ._llcrypt unregistered [ 4155.358922] Key type ._llcrypt registered [ 4155.360995] Key type .llcrypt registered [ 4155.463686] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 4157.601121] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4157.825856] alg: No test for adler32 (adler32-zlib) [ 4159.157846] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [ 4159.425647] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 4161.372771] Key type lgssc registered [ 4163.641304] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4175.562868] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4175.567132] LustreError: 104130:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 4175.591779] Lustre: Skipped 1 previous similar message [ 4175.591797] LustreError: 104129:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 4175.647261] LustreError: 104130:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 4175.651927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4175.658846] LustreError: 104130:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 4175.658876] LustreError: 104130:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 4177.887104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4188.132608] LustreError: 104167:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4188.162023] LustreError: 104167:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4191.197622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4191.628381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4197.743387] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4204.821910] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing lsmod [ 4208.659056] LustreError: 105334:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 4208.703172] LustreError: 105338:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 4208.713708] LustreError: 105334:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4955 [ 4208.721748] LustreError: 105334:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 4208.729879] LustreError: 105334:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 4208.737107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4209.289625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4214.761834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4214.772233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4214.779589] Lustre: Skipped 1 previous similar message [ 4214.807564] Lustre: Skipped 2 previous similar messages [ 4216.872201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4216.888797] Lustre: Skipped 2 previous similar messages [ 4219.886619] Lustre: server umount lustre-MDT0000 complete [ 4221.928751] LustreError: 104167:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4221.971327] LustreError: 104167:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4227.079194] LustreError: 104164:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4227.105484] LustreError: 104164:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4227.114233] LustreError: 104143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785781153 with bad export cookie 16873427611481409090 [ 4227.162816] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4227.231822] LustreError: 104599:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000645cd983 x1872527187008000/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4228.644707] Lustre: server umount lustre-MDT0001 complete [ 4246.635506] Lustre: server umount lustre-OST0000 complete [ 4259.216028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4259.820947] LustreError: 106406:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4259.961477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4265.449079] LustreError: 106407:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4266.909728] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4274.657791] LustreError: 106406:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4274.684281] LustreError: 106406:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4279.492191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4280.346918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4287.597483] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4299.975261] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4309.431084] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4318.840928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4319.391334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4328.224882] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4331.705616] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 4341.688955] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4352.486252] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4352.487847] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4352.494505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4352.511057] Lustre: Skipped 2 previous similar messages [ 4357.601613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4357.618726] Lustre: Skipped 2 previous similar messages [ 4358.127456] Lustre: server umount lustre-OST0000 complete [ 4367.329466] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4367.339832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4367.357471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4371.491144] Lustre: server umount lustre-MDT0000 complete [ 4372.967841] LustreError: 106406:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4373.001376] LustreError: 106406:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4377.010767] LustreError: 106387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785781303 with bad export cookie 16873427611481409790 [ 4377.047405] LustreError: 106387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4377.047522] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4378.284731] Lustre: server umount lustre-MDT0001 complete [ 4398.504989] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 4402.858778] Key type lgssc unregistered [ 4403.615542] LNet: 109967:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4403.637027] LNetError: 109967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4403.664924] LNet: Removed LNI 192.168.204.107@tcp [ 4405.466903] Key type .llcrypt unregistered [ 4405.470604] Key type ._llcrypt unregistered [ 4444.074914] Key type ._llcrypt registered [ 4444.076848] Key type .llcrypt registered [ 4444.197584] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 4445.688227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4445.939981] alg: No test for adler32 (adler32-zlib) [ 4447.272683] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [ 4447.737872] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 4449.560291] Key type lgssc registered [ 4451.439557] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4462.337512] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4462.376383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4464.184057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4470.243334] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4474.339261] LustreError: 111386:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4474.372968] LustreError: 111386:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4479.468402] LustreError: 111387:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4482.150310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4483.328760] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4491.912743] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4502.777585] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4513.382867] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4523.680988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4524.145661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4532.641921] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4535.425873] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 4547.930181] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4554.431582] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 4558.788724] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 4564.448455] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4564.455961] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4564.472138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4566.506318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4566.510479] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4566.535430] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4570.709895] Lustre: server umount lustre-OST0000 complete [ 4576.749579] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4576.755044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4576.774489] Lustre: Skipped 1 previous similar message [ 4576.797580] Lustre: Skipped 2 previous similar messages [ 4581.856816] LustreError: 111387:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4581.941450] Lustre: server umount lustre-MDT0000 complete [ 4587.008393] LustreError: 111387:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4587.042809] LustreError: 111387:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4589.327925] LustreError: 111367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785781515 with bad export cookie 13541386826935039111 [ 4589.328975] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4589.342189] LustreError: 111367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4590.437728] Lustre: server umount lustre-MDT0001 complete [ 4619.282348] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 4624.223044] Key type lgssc unregistered [ 4624.826958] LNet: 115052:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4624.837081] LNetError: 115052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4624.884698] LNet: Removed LNI 192.168.204.107@tcp [ 4626.342297] Key type .llcrypt unregistered [ 4626.346291] Key type ._llcrypt unregistered [ 4652.863541] Key type ._llcrypt registered [ 4652.869508] Key type .llcrypt registered [ 4653.109556] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 4655.132903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4655.327205] alg: No test for adler32 (adler32-zlib) [ 4656.845260] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [ 4657.260505] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 4658.976112] Key type lgssc registered [ 4660.701664] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4669.989645] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4670.014752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4671.613551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4676.705033] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4682.213904] LustreError: 116275:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4682.255784] LustreError: 116275:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4687.336677] LustreError: 116276:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4688.966389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4689.678963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4697.510597] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4708.807188] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4719.175024] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4730.625832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4731.230460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4739.336172] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4743.480714] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 4751.035747] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4769.253872] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4769.260654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4769.267409] Lustre: Skipped 1 previous similar message [ 4769.268458] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4769.286752] Lustre: Skipped 1 previous similar message [ 4769.664737] Lustre: server umount lustre-OST0000 complete [ 4776.417707] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4776.434637] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4776.473288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4779.489656] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4779.490278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4779.524016] Lustre: Skipped 1 previous similar message [ 4782.768562] Lustre: server umount lustre-MDT0000 complete [ 4784.620259] LustreError: 116275:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4784.688204] LustreError: 116275:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4788.706828] LustreError: 116276:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4788.757600] LustreError: 116276:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4789.067398] LustreError: 116256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785781715 with bad export cookie 1244730864382763380 [ 4789.094019] LustreError: 116256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4789.100368] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4790.042183] Lustre: server umount lustre-MDT0001 complete [ 4807.410337] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 4811.886081] Key type lgssc unregistered [ 4812.330125] LNet: 119835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4812.352656] LNetError: 119835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4813.420569] LNet: Removed LNI 192.168.204.107@tcp [ 4814.797811] Key type .llcrypt unregistered [ 4814.799239] Key type ._llcrypt unregistered [ 4850.702412] Key type ._llcrypt registered [ 4850.704726] Key type .llcrypt registered [ 4850.803987] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 4852.530649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4852.544895] alg: No test for adler32 (adler32-zlib) [ 4853.708919] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [ 4854.048979] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 4855.727130] Key type lgssc registered [ 4858.217429] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4869.933830] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4869.960549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4871.911330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4879.134414] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4882.409615] LustreError: 121271:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4882.451816] LustreError: 121271:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4887.520545] LustreError: 121272:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4892.646508] LustreError: 121271:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4894.174049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4894.803116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4901.938567] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4914.242113] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4923.064660] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4931.818361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4932.475548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4941.753098] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 4941.884526] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4953.093635] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4962.500130] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 4968.539984] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 4968.551633] Lustre: Skipped 2 previous similar messages [ 4969.132909] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 4969.191245] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 4976.124192] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 4976.136961] Lustre: Skipped 2 previous similar messages [ 4977.862017] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4977.869854] Lustre: Skipped 1 previous similar message [ 4989.926663] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.7@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 4989.941556] Lustre: Skipped 2 previous similar messages [ 4997.300136] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.7@tcp 192.168.0.[2,10]@tcp [ 5005.742365] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.7/24@tcp in log lustre-MDT0000 [ 5005.759365] Lustre: Skipped 2 previous similar messages [ 5014.651069] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.7/24@tcp [ 5014.668071] Lustre: Skipped 1 previous similar message [ 5026.828597] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5026.853308] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5026.928853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5028.840667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5028.847810] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5028.913933] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5030.526796] Lustre: server umount lustre-OST0000 complete [ 5038.060921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5038.073909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5038.091773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5038.099447] Lustre: Skipped 1 previous similar message [ 5044.192253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5044.210155] Lustre: Skipped 3 previous similar messages [ 5049.335212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5049.347045] Lustre: Skipped 1 previous similar message [ 5052.387091] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5053.277762] Lustre: server umount lustre-MDT0000 complete [ 5054.437744] LustreError: 121271:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5054.477641] LustreError: 121271:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5059.244418] LustreError: 121251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785781985 with bad export cookie 5249976131993970947 [ 5059.250812] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5059.261419] LustreError: 121251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5059.685208] Lustre: server umount lustre-MDT0001 complete [ 5084.851187] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 5089.750840] Key type lgssc unregistered [ 5090.529602] LNet: 126199:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5090.549533] LNetError: 126199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5090.596338] LNet: Removed LNI 192.168.204.107@tcp [ 5092.548851] Key type .llcrypt unregistered [ 5092.554207] Key type ._llcrypt unregistered [ 5130.090294] Key type ._llcrypt registered [ 5130.096968] Key type .llcrypt registered [ 5130.298346] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 5132.333089] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5132.616238] alg: No test for adler32 (adler32-zlib) [ 5134.068260] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [ 5134.510501] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 5136.311159] Key type lgssc registered [ 5138.430959] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5151.209880] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5151.233799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5152.970131] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 5153.018668] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.7/24@tcp [ 5153.022702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5159.266730] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5162.469503] LustreError: 127778:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5162.488915] LustreError: 127778:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5167.602736] LustreError: 127779:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5169.933178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5170.209242] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 5170.232846] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.7/24@tcp [ 5170.288638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5176.643019] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5191.299809] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5205.555866] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5219.128871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5219.692921] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5222.782090] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 5227.807062] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5238.822955] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5253.603232] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5253.619178] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5253.641399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5255.657843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5255.672581] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5255.682642] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5260.771724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5260.789329] Lustre: Skipped 1 previous similar message [ 5265.891490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5265.902085] Lustre: Skipped 2 previous similar messages [ 5266.399948] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5266.878178] Lustre: server umount lustre-OST0000 complete [ 5276.131634] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5276.136225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5276.154589] Lustre: Skipped 1 previous similar message [ 5276.179701] Lustre: Skipped 1 previous similar message [ 5279.939386] Lustre: server umount lustre-MDT0000 complete [ 5281.256275] LustreError: 127773:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5281.283120] LustreError: 127773:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5286.060081] LustreError: 130329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785782212 with bad export cookie 314034033286740200 [ 5286.064620] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5286.076885] LustreError: 130329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5287.101066] Lustre: server umount lustre-MDT0001 complete [ 5311.127909] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 5316.895491] Key type lgssc unregistered [ 5317.516671] LNet: 131396:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5317.560973] LNetError: 131396:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5317.603545] LNet: Removed LNI 192.168.204.107@tcp [ 5319.173781] Key type .llcrypt unregistered [ 5319.182023] Key type ._llcrypt unregistered [ 5355.838197] Key type ._llcrypt registered [ 5355.840284] Key type .llcrypt registered [ 5355.982495] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 5375.193162] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 5377.568822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5377.588878] alg: No test for adler32 (adler32-zlib) [ 5378.717954] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [ 5379.050980] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 5380.783156] Key type lgssc registered [ 5382.289098] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5438.822852] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5438.869593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5440.440828] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5440.522674] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5440.694737] Lustre: lustre-MDT0000: new disk, initializing [ 5440.862403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5440.914576] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5448.042577] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5462.345698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5462.538363] Lustre: 134804:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5462.656084] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5462.663964] Lustre: Skipped 1 previous similar message [ 5462.797505] Lustre: lustre-MDT0001: new disk, initializing [ 5462.954089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5463.048059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5463.069924] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5469.498051] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5482.091292] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5491.584249] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5504.220254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5504.681547] Lustre: lustre-OST0000: new disk, initializing [ 5504.685683] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5504.692160] Lustre: 136776:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5504.806983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5511.274976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5511.300933] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5511.484805] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5513.701967] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5528.497982] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5535.780274] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5536.121815] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5545.130933] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5545.517623] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5552.630393] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5552.635631] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5552.655359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5552.659296] Lustre: Skipped 1 previous similar message [ 5552.690809] LustreError: Skipped 1 previous similar message [ 5555.049746] Lustre: server umount lustre-OST0000 complete [ 5562.852728] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5562.882616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5562.886508] Lustre: Skipped 2 previous similar messages [ 5566.930815] Lustre: server umount lustre-MDT0000 complete [ 5572.459943] LustreError: 134798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785782499 with bad export cookie 5676397435530554231 [ 5572.462439] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5572.471534] LustreError: 134798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5573.027907] Lustre: server umount lustre-MDT0001 complete [ 5585.681951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5586.662355] LustreError: 138997:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5586.681727] LustreError: 138997:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5586.840235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5592.033525] LustreError: 138998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5595.906833] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5597.162507] LustreError: 138997:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5602.284613] LustreError: 138998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5607.392543] LustreError: 138997:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5608.354818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5608.549811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5614.565540] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5624.849192] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5632.412685] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5643.257670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5643.850275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5652.368765] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5665.692687] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5673.170752] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5673.478993] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5680.674631] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5681.074934] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5694.322056] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5694.602746] Lustre: lustre-OST0001: new disk, initializing [ 5694.617414] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5694.632167] Lustre: 142375:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5694.791939] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5701.244996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5701.257205] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5701.370138] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5702.650936] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5714.348819] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5714.607501] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5721.313673] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5721.662978] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5729.460382] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f72c892b800.ost_server_uuid 50 [ 5731.702433] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9f72c892b800.ost_server_uuid in IDLE state after 0 sec [ 5747.688834] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5747.695048] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5747.724532] Lustre: Skipped 1 previous similar message [ 5747.738013] LustreError: Skipped 1 previous similar message [ 5747.757091] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5747.768556] Lustre: Skipped 1 previous similar message [ 5752.802854] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5752.829114] Lustre: Skipped 1 previous similar message [ 5753.631960] Lustre: server umount lustre-OST0001 complete [ 5757.921297] LustreError: 143679:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5757.948464] LustreError: 143679:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5759.460936] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5759.481104] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5759.503727] Lustre: Skipped 1 previous similar message [ 5759.520062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5761.504266] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5765.714425] Lustre: server umount lustre-OST0000 complete [ 5772.256698] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5772.277744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5772.298921] Lustre: Skipped 1 previous similar message [ 5772.305419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5772.316874] Lustre: Skipped 3 previous similar messages [ 5778.401167] LustreError: 138993:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5778.437131] LustreError: 138993:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5778.503133] Lustre: server umount lustre-MDT0000 complete [ 5783.801732] LustreError: 140844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785782710 with bad export cookie 5676397435530555113 [ 5783.815019] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5783.818023] LustreError: 140844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5784.657915] Lustre: server umount lustre-MDT0001 complete [ 5817.733970] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 5823.364658] Key type lgssc unregistered [ 5823.823194] LNet: 145406:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5823.831136] LNetError: 145406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5823.850879] LNet: Removed LNI 192.168.204.107@tcp [ 5825.129402] Key type .llcrypt unregistered [ 5825.130733] Key type ._llcrypt unregistered [ 5889.503781] Key type ._llcrypt registered [ 5889.507205] Key type .llcrypt registered [ 5889.622530] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 5912.602985] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 5914.500372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5914.616027] alg: No test for adler32 (adler32-zlib) [ 5915.919048] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [ 5916.340861] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 5918.071157] Key type lgssc registered [ 5919.501445] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5967.798160] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5967.847030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5969.298142] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5969.336181] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5969.535095] Lustre: lustre-MDT0000: new disk, initializing [ 5969.659295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5969.684317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5975.758401] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5990.967823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5991.155811] Lustre: 149569:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5991.272512] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5991.275134] Lustre: Skipped 1 previous similar message [ 5991.408732] Lustre: lustre-MDT0001: new disk, initializing [ 5991.557691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5991.672336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5991.687579] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5997.934809] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6012.312413] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6019.027422] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6027.970525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6028.428219] Lustre: lustre-OST0000: new disk, initializing [ 6028.439702] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6028.450747] Lustre: 151539:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6028.629223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6037.640985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6037.647053] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6037.658503] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6037.829473] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6051.133866] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6058.183638] Lustre: Failing over lustre-OST0000 [ 6058.476212] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6058.479028] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6058.505668] Lustre: Skipped 1 previous similar message [ 6058.722815] Lustre: server umount lustre-OST0000 complete [ 6081.258544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6081.843536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6081.875128] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 6085.844892] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 6091.689867] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6097.706453] Lustre: Failing over lustre-MDT0000 [ 6098.409576] Lustre: server umount lustre-MDT0000 complete [ 6098.804077] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6098.822728] Lustre: Skipped 1 previous similar message [ 6098.893306] LustreError: 149574:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6098.908979] LustreError: 149574:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6101.253404] LustreError: 149575:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6104.033743] LustreError: 149580:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6104.070866] LustreError: 149580:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6106.323633] LustreError: 149575:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6111.440969] LustreError: 149574:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6111.474519] LustreError: 149574:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6115.295191] Lustre: 147789:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785783025/real 1785783025] req@00000000ab98b726 x1872529029383680/t0(0) o400->MGC192.168.204.107@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785783041 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6115.331217] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6120.430812] LustreError: 149576:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6120.448294] LustreError: 149576:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6120.868329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6124.547991] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x2308bd85acceb279 [ 6124.564767] Lustre: MGC192.168.204.107@tcp: Connection restored to 0@lo (at 0@lo) [ 6124.826433] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 6124.937243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6125.007962] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6127.821980] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6130.164317] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6130.185044] Lustre: lustre-OST0000: Recovery over after 0:45, of 3 clients 3 recovered and 0 were evicted. [ 6130.277342] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6130.532166] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6145.522305] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6145.522728] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6145.534988] Lustre: Skipped 1 previous similar message [ 6145.538417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6149.225434] Lustre: server umount lustre-OST0000 complete [ 6155.744994] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6155.747854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6155.756048] LustreError: Skipped 1 previous similar message [ 6155.761673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6155.780772] Lustre: Skipped 2 previous similar messages [ 6155.808569] Lustre: Skipped 2 previous similar messages [ 6160.872951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6160.887584] Lustre: Skipped 1 previous similar message [ 6161.178523] Lustre: server umount lustre-MDT0000 complete [ 6165.985083] LustreError: 149574:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6166.042483] LustreError: 149574:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6167.309484] LustreError: 149560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785783093 with bad export cookie 2524475972968690297 [ 6167.328482] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6167.975582] Lustre: server umount lustre-MDT0001 complete [ 6185.740700] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 6189.536613] Key type lgssc unregistered [ 6189.912508] LNet: 155208:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6189.923546] LNetError: 155208:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6189.938201] LNet: Removed LNI 192.168.204.107@tcp [ 6191.146977] Key type .llcrypt unregistered [ 6191.148485] Key type ._llcrypt unregistered [ 6246.392325] Key type ._llcrypt registered [ 6246.396600] Key type .llcrypt registered [ 6246.554471] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 6247.834142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6247.895544] alg: No test for adler32 (adler32-zlib) [ 6249.204956] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [ 6249.573339] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 6251.329379] Key type lgssc registered [ 6252.942141] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6265.113830] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6265.165343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6266.942549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6274.432382] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6277.089656] LustreError: 156732:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6277.132477] LustreError: 156732:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6282.211595] LustreError: 156733:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6287.333380] LustreError: 156732:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6292.317908] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6292.454589] LustreError: 156733:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6297.579418] LustreError: 156732:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6298.821645] LustreError: 156761:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6298.852816] LustreError: 156761:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6298.870626] LustreError: 156761:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 32, retries 0, failed: rc = -5 [ 6299.817954] Lustre: server umount lustre-MDT0000 complete [ 6323.827351] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 6340.889595] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 6396.770954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6397.093843] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6397.129472] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6397.230119] Lustre: lustre-MDT0000: new disk, initializing [ 6397.314611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6397.336043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6404.312316] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6419.252623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6419.567578] Lustre: 160091:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6419.695364] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6419.702689] Lustre: Skipped 1 previous similar message [ 6419.820437] Lustre: lustre-MDT0001: new disk, initializing [ 6419.975225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6420.036152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6420.045030] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6427.810486] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6442.033875] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6450.131941] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6460.136568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6460.628691] Lustre: lustre-OST0000: new disk, initializing [ 6460.635917] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6460.645290] Lustre: 162064:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6460.759331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6467.157384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6467.172527] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6467.319333] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6469.500815] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6487.545986] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6492.901573] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 6495.977099] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 14:56:59 (1785783419) [ 6880.223728] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6880.227216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6880.281866] Lustre: Skipped 1 previous similar message [ 6882.887439] Lustre: server umount lustre-MDT0000 complete [ 6885.344081] LustreError: 160097:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6885.368315] LustreError: 160097:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6888.606641] LustreError: 160084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785783815 with bad export cookie 9361378674704821770 [ 6888.615283] LustreError: 160084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6888.617030] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6890.463912] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6890.496031] Lustre: Skipped 1 previous similar message [ 6890.506793] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6890.511979] Lustre: Skipped 2 previous similar messages [ 6893.472152] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6895.748382] Lustre: server umount lustre-MDT0001 complete [ 6908.456905] Lustre: server umount lustre-OST0000 complete [ 6920.295852] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 6933.910288] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 6986.966433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6987.464228] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6987.509887] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6987.646401] Lustre: lustre-MDT0000: new disk, initializing [ 6987.801835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6987.836805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6996.049436] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7012.347220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7012.702485] Lustre: 166762:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7012.854477] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7012.869669] Lustre: Skipped 1 previous similar message [ 7013.044623] Lustre: lustre-MDT0001: new disk, initializing [ 7013.251450] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7013.325636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7013.343298] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7020.407150] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7036.508584] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7045.204339] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7055.660452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7056.197306] Lustre: lustre-OST0000: new disk, initializing [ 7056.209048] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7056.224077] Lustre: 168735:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7056.378956] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7062.146953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7062.165502] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7062.315367] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7064.851483] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7078.110092] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7085.929750] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7086.634461] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7095.059890] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7095.351509] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7098.341776] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7098.346791] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7098.351710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7098.367189] Lustre: Skipped 1 previous similar message [ 7098.375023] LustreError: Skipped 1 previous similar message [ 7103.478050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7103.492340] Lustre: Skipped 2 previous similar messages [ 7104.733888] Lustre: server umount lustre-OST0000 complete [ 7110.624102] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7110.641630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7113.699780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7113.710286] Lustre: Skipped 1 previous similar message [ 7116.796141] Lustre: server umount lustre-MDT0000 complete [ 7118.821234] LustreError: 169199:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7118.839956] LustreError: 169199:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7121.371706] LustreError: 167569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785784047 with bad export cookie 9361378674704980607 [ 7121.374055] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7121.392521] LustreError: 167569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7122.358645] Lustre: server umount lustre-MDT0001 complete [ 7135.060200] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 15:07:38 (1785784058) [ 7157.493552] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 7169.316855] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 7215.502558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7215.848603] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7215.921286] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7216.046820] Lustre: lustre-MDT0000: new disk, initializing [ 7216.114030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7216.133465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7221.806360] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7238.269632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7238.486889] Lustre: 173495:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7238.963932] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7244.627188] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7257.368557] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7266.300704] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7275.468228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7276.026708] Lustre: lustre-OST0000: new disk, initializing [ 7276.032533] Lustre: Skipped 1 previous similar message [ 7276.037065] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7276.040964] Lustre: Skipped 2 previous similar messages [ 7276.058340] Lustre: 175478:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7284.847231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7284.867435] Lustre: Skipped 1 previous similar message [ 7284.877223] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7285.030578] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7285.213816] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7299.694399] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7315.935892] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7315.938085] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7315.952094] LustreError: Skipped 1 previous similar message [ 7315.979663] Lustre: Skipped 1 previous similar message [ 7315.995336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7316.016844] Lustre: Skipped 1 previous similar message [ 7319.534560] Lustre: server umount lustre-OST0000 complete [ 7326.191915] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7326.196295] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7326.230731] Lustre: Skipped 2 previous similar messages [ 7331.296856] LustreError: 173508:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7331.304787] LustreError: 173508:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7331.388972] Lustre: server umount lustre-MDT0000 complete [ 7335.288459] LustreError: 173489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785784261 with bad export cookie 9361378674704981489 [ 7335.294171] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7335.311287] LustreError: 173489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7336.130908] Lustre: server umount lustre-MDT0001 complete [ 7347.313886] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 15:11:11 (1785784271) [ 7367.162923] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 7384.280791] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 7431.551726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7431.846973] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7431.877906] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7431.968675] Lustre: lustre-MDT0000: new disk, initializing [ 7432.083498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7432.088595] Lustre: Skipped 2 previous similar messages [ 7432.100742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7438.291412] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7450.645130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7450.831953] Lustre: 179823:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7451.335980] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7457.341657] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7471.725884] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7481.836315] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7494.920840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7495.645992] Lustre: 181805:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7501.414157] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7501.645673] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7506.950312] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7521.126907] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7532.516286] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7532.520846] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7532.527296] LustreError: Skipped 1 previous similar message [ 7532.563142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7532.566287] Lustre: Skipped 1 previous similar message [ 7532.628197] Lustre: Skipped 3 previous similar messages [ 7533.609978] Lustre: server umount lustre-OST0000 complete [ 7547.232081] Lustre: server umount lustre-MDT0000 complete [ 7547.884211] LustreError: 179830:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7547.948129] LustreError: 179830:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7555.817394] LustreError: 179816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785784482 with bad export cookie 9361378674704982574 [ 7555.825810] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7555.837486] LustreError: 179816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7556.347714] Lustre: server umount lustre-MDT0001 complete [ 7577.559247] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 7581.803867] Key type lgssc unregistered [ 7582.282201] LNet: 183685:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7582.291667] LNetError: 183685:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7582.307926] LNet: Removed LNI 192.168.204.107@tcp [ 7583.512155] Key type .llcrypt unregistered [ 7583.519372] Key type ._llcrypt unregistered [ 7616.908423] Key type ._llcrypt registered [ 7616.914159] Key type .llcrypt registered [ 7617.047439] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 7618.378544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7618.806262] alg: No test for adler32 (adler32-zlib) [ 7619.903939] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [ 7620.162038] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 7621.831184] Key type lgssc registered [ 7623.408804] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7632.139366] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7632.161373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7633.836919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7640.710816] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7644.130725] LustreError: 185107:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7644.154623] LustreError: 185107:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7649.269934] LustreError: 185108:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7651.324824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7651.735087] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7657.467163] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7667.894767] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7676.135499] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7684.913283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7685.507856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7693.536398] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7702.818289] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7704.947275] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 7730.666799] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7730.669993] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7730.676568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7730.706831] Lustre: Skipped 1 previous similar message [ 7735.779044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7735.794348] Lustre: Skipped 2 previous similar messages [ 7736.392975] Lustre: server umount lustre-OST0000 complete [ 7743.968485] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7743.977704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7743.999855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7746.017862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7746.020197] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7748.702556] Lustre: server umount lustre-MDT0000 complete [ 7751.163070] LustreError: 185108:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7751.196775] LustreError: 185108:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7754.507564] LustreError: 185088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785784681 with bad export cookie 17189770181224670472 [ 7754.511439] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7754.515820] LustreError: 185088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7755.272683] Lustre: server umount lustre-MDT0001 complete [ 7770.366573] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 7774.336635] Key type lgssc unregistered [ 7774.729473] LNet: 188665:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7774.738583] LNetError: 188665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7775.793906] LNet: Removed LNI 192.168.204.107@tcp [ 7777.101463] Key type .llcrypt unregistered [ 7777.103178] Key type ._llcrypt unregistered [ 7808.432437] Key type ._llcrypt registered [ 7808.437246] Key type .llcrypt registered [ 7808.584744] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 7810.045191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7810.293040] alg: No test for adler32 (adler32-zlib) [ 7811.503668] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [ 7811.908406] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 7813.639187] Key type lgssc registered [ 7815.263908] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7823.290639] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7823.316952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7824.806293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7830.524069] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7835.108591] LustreError: 190085:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7835.125722] LustreError: 190085:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7840.235156] LustreError: 190086:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7840.991472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7841.611434] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7847.113812] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7857.295329] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7865.031923] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7873.660301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7874.095125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7881.298577] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7890.257963] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7894.578300] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7902.690264] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7902.698970] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7902.718356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7904.754884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7904.763849] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7904.775840] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7906.542252] Lustre: lustre-OST0000: Not available for connect from 192.168.204.7@tcp (stopping) [ 7906.548868] Lustre: Skipped 1 previous similar message [ 7908.361268] Lustre: server umount lustre-OST0000 complete [ 7917.063281] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9f72c373c800.ost_server_uuid 50 [ 7919.040992] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 7935.456656] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7935.460189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7935.477349] Lustre: Skipped 1 previous similar message [ 7940.576355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7940.585841] Lustre: Skipped 2 previous similar messages [ 7942.123941] Lustre: server umount lustre-MDT0000 complete [ 7945.700413] LustreError: 190085:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7945.730777] LustreError: 190085:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7947.088205] LustreError: 190067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785784873 with bad export cookie 12191811586881065640 [ 7947.096177] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7947.107674] LustreError: 190067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7947.623605] Lustre: server umount lustre-MDT0001 complete [ 7957.912904] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 15:21:21 (1785784881) [ 7965.365032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7966.158237] LustreError: 194006:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7966.270201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7971.306256] LustreError: 194007:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7971.337468] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7981.182900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7981.603836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7986.635249] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7994.230474] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8001.461389] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8009.438333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8009.740547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8010.802484] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8016.247321] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8025.417411] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8033.114738] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8033.424381] Lustre: lustre-OST0001: new disk, initializing [ 8033.432486] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8033.441330] Lustre: 196846:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8033.586607] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8040.496258] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8042.108498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8042.120278] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8042.225268] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8052.154175] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8056.804918] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8056.813857] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8056.839540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8062.205768] Lustre: server umount lustre-OST0000 complete [ 8062.955224] LustreError: 195830:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8062.980790] LustreError: 195830:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8069.898734] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8070.078165] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 8074.868876] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8075.027677] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 8083.438931] LustreError: 195830:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8083.466715] LustreError: 195830:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 8093.665513] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8093.669457] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8093.672390] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8093.672397] Lustre: Skipped 3 previous similar messages [ 8093.681305] LustreError: Skipped 1 previous similar message [ 8093.729185] Lustre: Skipped 2 previous similar messages [ 8096.789551] Lustre: server umount lustre-OST0001 complete [ 8103.904852] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8107.680360] Lustre: server umount lustre-MDT0000 complete [ 8112.342428] LustreError: 193987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785785038 with bad export cookie 12191811586881066711 [ 8112.349179] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8112.356915] LustreError: 193987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8113.028621] Lustre: server umount lustre-MDT0001 complete [ 8140.078892] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 15:24:24 (1785785064) [ 8147.662462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8148.109629] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8148.185414] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8148.502944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8153.932205] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8163.649377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8163.788054] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8163.845059] Lustre: 200284:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8168.801156] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8176.361524] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8182.956465] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8190.158050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8190.347027] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8197.908972] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8198.818339] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 8206.430959] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8213.877910] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8213.961169] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8214.187895] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8214.191752] Lustre: Skipped 2 previous similar messages [ 8222.067166] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8223.541514] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 8232.284984] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8238.346318] Lustre: lustre-OST0000: Not available for connect from 192.168.204.7@tcp (stopping) [ 8238.353573] Lustre: Skipped 3 previous similar messages [ 8239.079887] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8239.106503] Lustre: Skipped 2 previous similar messages [ 8241.613158] Lustre: server umount lustre-OST0000 complete [ 8243.408120] LustreError: 202554:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8243.442907] LustreError: 202554:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 8264.687564] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8264.710061] LustreError: Skipped 2 previous similar messages [ 8268.209358] Lustre: server umount lustre-OST0001 complete [ 8274.915973] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8274.931188] Lustre: Skipped 3 previous similar messages [ 8280.269986] Lustre: server umount lustre-MDT0000 complete [ 8284.985561] LustreError: 201040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785785211 with bad export cookie 12191811586881068118 [ 8284.995248] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8285.001702] LustreError: 201040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8285.035273] LustreError: 189396:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting [ 8285.466423] Lustre: server umount lustre-MDT0001 complete [ 8314.487809] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 15:27:18 (1785785238) [ 8342.100941] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 8371.178733] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 8425.850963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8426.472779] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8426.547221] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8426.674639] Lustre: lustre-MDT0000: new disk, initializing [ 8426.791658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8426.822633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8432.450240] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8451.757407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8452.091540] Lustre: 208410:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8452.233675] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8452.245987] Lustre: Skipped 1 previous similar message [ 8452.475261] Lustre: lustre-MDT0001: new disk, initializing [ 8452.828236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8452.843742] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8459.268153] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8474.341473] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8483.444744] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8493.859114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8494.248796] Lustre: lustre-OST0000: new disk, initializing [ 8494.261134] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8494.283730] Lustre: 210381:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8499.870572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8499.882964] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8500.016862] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8502.774603] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8515.332251] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8522.001818] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8522.253344] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8529.493410] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8529.852906] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8536.040440] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8536.046217] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8536.068906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8536.068913] Lustre: Skipped 6 previous similar messages [ 8538.479461] Lustre: server umount lustre-OST0000 complete [ 8550.415470] Lustre: server umount lustre-MDT0000 complete [ 8551.401374] LustreError: 208422:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8551.426520] LustreError: 208422:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 8554.508925] LustreError: 210391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785785481 with bad export cookie 12191811586881069903 [ 8554.522092] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8554.528135] LustreError: 210391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8554.910357] Lustre: server umount lustre-MDT0001 complete [ 8566.740583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8574.952905] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8587.647673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8593.931180] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8603.265569] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8612.931107] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8622.692031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8632.031847] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8642.438911] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8650.292949] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8650.798932] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8658.576155] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8659.039658] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8662.505137] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8662.521183] LustreError: Skipped 2 previous similar messages [ 8667.499445] Lustre: server umount lustre-OST0000 complete [ 8676.039953] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8681.414715] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8713.286238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8713.834707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8713.844187] Lustre: Skipped 5 previous similar messages [ 8721.613600] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8729.189760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8729.213428] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8729.216048] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8731.663268] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8733.362728] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8739.812111] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8739.816883] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8739.834566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8739.853364] Lustre: Skipped 6 previous similar messages [ 8739.881974] Lustre: Skipped 7 previous similar messages [ 8744.882748] Lustre: server umount lustre-OST0000 complete [ 8756.416974] Lustre: server umount lustre-MDT0000 complete [ 8760.288915] LustreError: 212595:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8760.296739] LustreError: 212595:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 8762.688681] LustreError: 212582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785785689 with bad export cookie 12191811586881070785 [ 8762.696153] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8762.705502] LustreError: 212582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8763.963835] Lustre: server umount lustre-MDT0001 complete [ 8777.484510] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 15:35:01 (1785785701) [ 8787.715534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8794.442526] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8805.328089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8810.305786] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8820.878639] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8828.597321] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8838.360364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8846.413935] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8855.421890] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8860.523791] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8860.696260] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8867.311631] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8867.655506] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8876.060405] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8876.293503] Lustre: lustre-OST0001: new disk, initializing [ 8876.299835] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8876.310536] Lustre: 221947:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8883.578021] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8883.898569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8883.911451] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8884.073265] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8896.084813] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8902.740368] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 8902.922651] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8909.788598] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 8910.098646] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8914.918455] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8914.932234] LustreError: Skipped 1 previous similar message [ 8927.199138] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8927.550391] Lustre: server umount lustre-OST0001 complete [ 8939.577120] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 8939.924928] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 8950.522816] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 8951.138506] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 8983.543946] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8989.167846] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 8989.194518] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 8989.198555] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8989.220989] Lustre: Skipped 1 previous similar message [ 8992.612863] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9003.869253] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9006.078920] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9009.671122] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9009.680662] Lustre: Skipped 4 previous similar messages [ 9009.689529] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9009.697775] Lustre: Skipped 11 previous similar messages [ 9015.138500] Lustre: server umount lustre-OST0001 complete [ 9019.889171] LustreError: 220397:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9019.916985] LustreError: 220397:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 34 previous similar messages [ 9041.332977] Lustre: server umount lustre-OST0000 complete [ 9054.777294] Lustre: server umount lustre-MDT0000 complete [ 9060.207723] LustreError: 218558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785785986 with bad export cookie 12191811586881071751 [ 9060.216789] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9060.226455] LustreError: 218558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9066.726027] Lustre: server umount lustre-MDT0001 complete [ 9098.316761] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 15:40:21 (1785786021) [ 9109.182408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9109.633706] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9109.689928] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9116.954223] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9129.789478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9130.031240] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9130.063810] Lustre: 227203:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9137.582806] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9149.806657] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9157.706657] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9166.065292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9166.419476] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9174.981497] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9185.379964] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9198.670254] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9198.788733] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9207.191028] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9217.410883] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9219.854941] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9227.571861] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 9227.741713] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9241.017261] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 9241.797137] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9262.090548] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f72c70fe000.ost_server_uuid 50 [ 9266.929080] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9f72c70fe000.ost_server_uuid in IDLE state after 0 sec [ 9268.335410] Lustre: Permanently deactivating lustre-OST0001 [ 9268.356315] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 9274.004853] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 9274.266266] Lustre: Permanently reactivating lustre-OST0001 [ 9274.283753] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 9274.291203] Lustre: Skipped 2 previous similar messages [ 9287.992812] Lustre: server umount lustre-OST0001 complete [ 9293.792857] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9293.800984] LustreError: Skipped 6 previous similar messages [ 9299.613113] Lustre: server umount lustre-OST0000 complete [ 9302.495657] LustreError: 231787:0:(osp_dev.c:699:osp_process_config()) lustre-OST0001-osc-MDT0001: unknown param osc.active=1 [ 9322.975136] LustreError: 232102:0:(osp_dev.c:699:osp_process_config()) lustre-OST0001-osc-MDT0000: unknown param osc.active=1 [ 9337.312027] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9337.704261] Lustre: server umount lustre-MDT0000 complete [ 9342.507466] LustreError: 229067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785786269 with bad export cookie 12191811586881073039 [ 9342.515170] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9342.523694] LustreError: 229067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9343.193632] Lustre: server umount lustre-MDT0001 complete [ 9378.220637] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 15:45:01 (1785786301) [ 9390.205030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9390.672055] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9390.786429] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9390.799336] Lustre: Skipped 2 previous similar messages [ 9391.159705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9391.168343] Lustre: Skipped 9 previous similar messages [ 9398.140436] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9410.462742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9410.706653] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9410.776697] Lustre: 233922:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9416.499117] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9427.098448] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9436.334904] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9445.686937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9446.067661] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9446.115104] Lustre: Permanently deactivating lustre-OST0000 [ 9446.131087] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 9453.959926] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9455.220772] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9455.232910] Lustre: Skipped 1 previous similar message [ 9455.311881] LustreError: 236219:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -108 [ 9463.142504] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9472.375027] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9472.559870] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9479.524771] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 9482.505354] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9492.337732] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9495.015621] Lustre: Permanently reactivating lustre-OST0000 [ 9495.048496] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 9495.051184] Lustre: Skipped 2 previous similar messages [ 9501.318613] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 9501.319545] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 46 seconds. I think it's dead, and I am evicting it. exp 000000001786766d, cur 1785786427 deadline 1785786411 last 1785786381 [ 9501.332766] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 9501.367443] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9501.383968] Lustre: Skipped 1 previous similar message [ 9501.476487] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 9525.732814] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9525.755793] Lustre: Skipped 28 previous similar messages [ 9527.263203] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9527.359974] Lustre: server umount lustre-OST0001 complete [ 9532.386477] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9532.418333] Lustre: Skipped 15 previous similar messages [ 9535.983044] LustreError: 235772:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9536.002402] LustreError: 235772:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 9537.973856] Lustre: server umount lustre-OST0000 complete [ 9550.603623] Lustre: server umount lustre-MDT0000 complete [ 9555.937315] LustreError: 237519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785786482 with bad export cookie 12191811586881074803 [ 9555.950076] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9555.952051] LustreError: 237519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9557.141753] Lustre: server umount lustre-MDT0001 complete [ 9574.880792] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 9579.369572] Key type lgssc unregistered [ 9579.815192] LNet: 238811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9579.831889] LNetError: 238811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9579.850914] LNet: Removed LNI 192.168.204.107@tcp [ 9580.771370] Key type .llcrypt unregistered [ 9580.773653] Key type ._llcrypt unregistered [ 9612.912723] Key type ._llcrypt registered [ 9612.917341] Key type .llcrypt registered [ 9613.110937] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 9615.648816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9615.705617] alg: No test for adler32 (adler32-zlib) [ 9617.019652] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [ 9617.405920] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 9619.199361] Key type lgssc registered [ 9620.640247] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9634.540734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9634.582904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9636.544861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9645.453308] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9646.069766] LustreError: 240289:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9646.091677] LustreError: 240289:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9651.174160] LustreError: 240290:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9656.292058] LustreError: 240289:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9659.996748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9660.158663] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9660.209546] Lustre: 240276:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9660.225877] Lustre: Permanently deactivating lustre-MDT0001 [ 9660.239535] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 9660.737736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9666.843671] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9670.178311] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 9678.593908] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9687.579729] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9697.623743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9698.069264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9705.915825] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9706.371668] LustreError: 242134:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9706.407574] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 9707.466899] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 9712.620974] LustreError: 242551:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9712.643292] LustreError: 242551:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9717.029861] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9722.855965] LustreError: 242134:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9722.889727] LustreError: 242134:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9725.928638] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9726.437284] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9731.580633] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 9731.595802] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 9734.317210] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9744.801746] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9748.571436] Lustre: Permanently reactivating lustre-MDT0001 [ 9748.615799] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 9748.621595] Lustre: Skipped 2 previous similar messages [ 9750.885475] Lustre: lustre-MDT0001: Client 52b3658d-f631-4350-a47a-acc8bb0627fc (at 192.168.204.7@tcp) reconnecting [ 9755.324137] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9755.339470] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 9755.352399] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 9755.381301] LustreError: 244003:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 9755.392433] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 9755.395718] LustreError: 244003:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 9755.395843] LustreError: 244003:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.107@tcp: cfg command failed: rc = -108 [ 9755.400067] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 9755.400067] [ 9755.436168] LustreError: 240281:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 9758.243054] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9760.880517] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9770.202459] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 9770.605459] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 9775.784350] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 9775.936125] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 9779.359217] Lustre: Permanently deactivating lustre-MDT0001 [ 9779.391329] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 9779.400599] Lustre: Skipped 2 previous similar messages [ 9785.651487] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 9798.115156] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9798.119778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9798.140622] Lustre: Skipped 2 previous similar messages [ 9798.162247] Lustre: Skipped 3 previous similar messages [ 9802.582581] Lustre: server umount lustre-MDT0000 complete [ 9803.248533] LustreError: 240290:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9803.273511] LustreError: 240290:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9807.055722] LustreError: 240269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785786733 with bad export cookie 5015114263213931502 [ 9807.056747] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9807.067664] LustreError: 240269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9807.787514] Lustre: server umount lustre-MDT0001 complete [ 9815.071691] Lustre: server umount lustre-OST0000 complete [ 9831.783077] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 15:52:35 (1785786755) [ 9847.820671] Lustre: server umount lustre-OST0001 complete [ 9858.081039] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 9872.837654] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 9925.205157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9925.864436] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9925.895804] Lustre: Skipped 2 previous similar messages [ 9925.933692] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9926.190427] Lustre: lustre-MDT0000: new disk, initializing [ 9926.293227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9926.310790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9935.538883] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9953.945082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9954.183557] Lustre: 248572:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9954.290764] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9954.303214] Lustre: Skipped 1 previous similar message [ 9954.531745] Lustre: lustre-MDT0001: new disk, initializing [ 9954.780479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9954.898338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9954.923161] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9961.244676] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9975.112645] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9984.411549] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9993.557746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9993.995230] Lustre: lustre-OST0000: new disk, initializing [ 9994.004564] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9994.021696] Lustre: 250544:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9994.161123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10002.478376] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10002.509708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10002.519731] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10002.625454] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10020.550362] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10024.056749] LustreError: 248580:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [10034.106193] LustreError: 248580:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [10034.157700] LustreError: 249993:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [10040.185283] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10040.883434] Lustre: lustre-OST0001: new disk, initializing [10040.896477] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10040.919254] Lustre: 251664:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10041.207497] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10044.264360] LustreError: 249993:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [10049.850141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10049.870770] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10050.102312] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10053.701842] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10067.817178] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10070.603665] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [10075.632160] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10075.633760] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10075.652609] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10075.670785] Lustre: Skipped 2 previous similar messages [10075.707949] LustreError: Skipped 1 previous similar message [10080.057819] Lustre: server umount lustre-OST0001 complete [10080.749452] LustreError: 250910:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10080.786260] LustreError: 250910:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10105.312434] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10105.312659] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10105.333631] LustreError: Skipped 1 previous similar message [10105.362405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10105.380061] Lustre: Skipped 1 previous similar message [10107.721105] Lustre: server umount lustre-OST0000 complete [10113.505378] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10113.514760] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10113.537747] Lustre: Skipped 1 previous similar message [10113.545966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10113.548298] Lustre: Skipped 2 previous similar messages [10120.156298] Lustre: server umount lustre-MDT0000 complete [10125.884069] LustreError: 248565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785787052 with bad export cookie 5015114263213934316 [10125.888908] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10125.897169] LustreError: 248565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10132.496807] Lustre: server umount lustre-MDT0001 complete [10162.362969] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [10166.887453] Key type lgssc unregistered [10167.362801] LNet: 253722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10167.376691] LNetError: 253722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10167.401582] LNet: Removed LNI 192.168.204.107@tcp [10169.200822] Key type .llcrypt unregistered [10169.207257] Key type ._llcrypt unregistered [10226.402822] Key type ._llcrypt registered [10226.408027] Key type .llcrypt registered [10226.536196] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [10228.247995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10228.468871] alg: No test for adler32 (adler32-zlib) [10229.773311] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [10230.070332] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [10231.903184] Key type lgssc registered [10233.683886] Lustre: Echo OBD driver; http://www.lustre.org/ [10243.198869] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10243.240787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10244.494301] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10244.525700] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10244.719951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10251.193716] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10262.196322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10262.396445] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10262.468104] Lustre: 255936:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10262.871969] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10269.655385] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10282.182234] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10290.788547] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10299.519905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10299.855466] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10300.189243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10308.810063] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [10309.593213] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10321.462638] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10385.896662] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10385.900074] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10385.900850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10385.910020] LustreError: Skipped 1 previous similar message [10385.952728] Lustre: Skipped 1 previous similar message [10391.359275] Lustre: server umount lustre-OST0000 complete [10396.406465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [10411.803330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10412.089393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10416.166415] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [10416.192751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [10416.194467] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [10416.224556] Lustre: Skipped 1 previous similar message [10419.645295] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10429.700225] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10472.424434] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10472.438336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10472.439602] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10472.443030] Lustre: Skipped 1 previous similar message [10474.566210] Lustre: server umount lustre-OST0000 complete [10482.658995] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10482.674833] Lustre: Skipped 2 previous similar messages [10482.678524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10482.692189] Lustre: Skipped 2 previous similar messages [10486.761975] Lustre: server umount lustre-MDT0000 complete [10487.779981] LustreError: 257534:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10487.811470] LustreError: 257534:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10491.920656] LustreError: 255927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785787418 with bad export cookie 8215615449423362843 [10491.924125] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10491.940149] LustreError: 255927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10492.629483] Lustre: server umount lustre-MDT0001 complete [10511.031617] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [10514.600235] Key type lgssc unregistered [10514.883102] LNet: 261400:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10514.904715] LNetError: 261400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10514.919960] LNet: Removed LNI 192.168.204.107@tcp [10516.076479] Key type .llcrypt unregistered [10516.078285] Key type ._llcrypt unregistered [10548.948405] Key type ._llcrypt registered [10548.959186] Key type .llcrypt registered [10549.138829] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [10551.285934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10551.313466] alg: No test for adler32 (adler32-zlib) [10552.530227] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [10552.964065] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [10554.791188] Key type lgssc registered [10556.329752] Lustre: Echo OBD driver; http://www.lustre.org/ [10567.547032] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10567.600063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10569.148489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10575.821738] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10579.430385] LustreError: 262822:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10579.452069] LustreError: 262822:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10584.562208] LustreError: 262823:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10589.674254] LustreError: 262822:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10591.311944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10592.237200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10599.170180] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10609.330450] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10618.272557] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10628.272867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10628.872371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10638.110489] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10639.092169] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [10649.078146] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10683.360965] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10683.371281] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10683.401305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10685.413969] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10685.423877] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10685.430179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10687.851518] Lustre: server umount lustre-OST0000 complete [10694.630059] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10694.644295] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10694.656808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10694.664247] Lustre: Skipped 1 previous similar message [10699.741613] Lustre: server umount lustre-MDT0000 complete [10700.775173] LustreError: 262823:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10700.793278] LustreError: 262823:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10705.092937] LustreError: 262801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785787631 with bad export cookie 7851160984756608317 [10705.108441] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10706.115428] Lustre: server umount lustre-MDT0001 complete [10722.693873] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [10726.540554] Key type lgssc unregistered [10726.954946] LNet: 267134:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10726.968558] LNetError: 267134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10726.992814] LNet: Removed LNI 192.168.204.107@tcp [10728.004635] Key type .llcrypt unregistered [10728.009371] Key type ._llcrypt unregistered [10753.889838] Key type ._llcrypt registered [10753.891154] Key type .llcrypt registered [10754.040695] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [10755.730788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10755.808062] alg: No test for adler32 (adler32-zlib) [10756.975297] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [10757.388290] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [10759.208553] Key type lgssc registered [10760.712043] Lustre: Echo OBD driver; http://www.lustre.org/ [10770.700587] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10770.736412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10772.391072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10778.297927] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10781.747545] LustreError: 268359:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10781.794911] LustreError: 268359:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10786.801857] LustreError: 268360:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10791.924804] LustreError: 268359:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10792.877101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10793.147363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10805.516363] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10826.354098] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10840.791732] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10849.715201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10850.128088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10860.174079] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10866.575309] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10873.281166] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10892.257175] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10892.276649] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10892.301511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10892.313509] Lustre: Skipped 1 previous similar message [10896.631883] Lustre: server umount lustre-OST0000 complete [10902.505736] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10902.528204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10902.530975] Lustre: Skipped 2 previous similar messages [10902.554539] Lustre: Skipped 1 previous similar message [10907.618227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10908.992316] Lustre: server umount lustre-MDT0000 complete [10911.718340] LustreError: 270812:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10911.732848] LustreError: 270812:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10913.424703] LustreError: 269097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785787839 with bad export cookie 1510994282484203815 [10913.431497] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10913.455558] LustreError: 269097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10914.389343] Lustre: server umount lustre-MDT0001 complete [10929.834912] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [10933.531179] Key type lgssc unregistered [10934.045750] LNet: 272049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10934.061532] LNetError: 272049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10934.077918] LNet: Removed LNI 192.168.204.107@tcp [10935.200242] Key type .llcrypt unregistered [10935.201951] Key type ._llcrypt unregistered [10974.511295] Key type ._llcrypt registered [10974.514730] Key type .llcrypt registered [10974.730610] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [10976.437698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10976.763464] alg: No test for adler32 (adler32-zlib) [10978.225680] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [10978.763029] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [10980.503220] Key type lgssc registered [10982.542679] Lustre: Echo OBD driver; http://www.lustre.org/ [10995.492550] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10995.521823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10997.587161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11003.447519] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11006.961619] LustreError: 273469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11006.992283] LustreError: 273469:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11012.073689] LustreError: 273470:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11017.184842] LustreError: 273469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11017.378567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11018.197417] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11023.902355] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11036.097057] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11048.486059] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11059.685804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11060.396596] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11066.560618] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [11069.990121] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11081.418251] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11114.985131] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11115.006436] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11115.032805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11118.050362] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11118.052363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11118.073601] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11118.114602] Lustre: Skipped 1 previous similar message [11120.065746] Lustre: server umount lustre-OST0000 complete [11125.744286] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11125.759111] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11125.788230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11128.292631] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11128.302927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11132.034713] Lustre: server umount lustre-MDT0000 complete [11133.409935] LustreError: 273466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11133.430308] LustreError: 273466:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11137.032913] LustreError: 273449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785788063 with bad export cookie 16708039457460581109 [11137.040412] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11137.046149] LustreError: 273449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11137.694419] Lustre: server umount lustre-MDT0001 complete [11155.920520] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [11159.789331] Key type lgssc unregistered [11160.209543] LNet: 277795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11160.215947] LNetError: 277795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11160.245645] LNet: Removed LNI 192.168.204.107@tcp [11161.349818] Key type .llcrypt unregistered [11161.353508] Key type ._llcrypt unregistered [11188.875980] Key type ._llcrypt registered [11188.879822] Key type .llcrypt registered [11189.068480] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [11191.492131] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11191.650996] alg: No test for adler32 (adler32-zlib) [11192.831890] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [11193.158805] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [11194.823247] Key type lgssc registered [11197.094151] Lustre: Echo OBD driver; http://www.lustre.org/ [11207.250835] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11207.307399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11208.925179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11215.139519] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11219.431283] LustreError: 279036:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11219.467109] LustreError: 279036:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11224.547654] LustreError: 279038:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11228.672192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11229.510855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11238.114799] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11249.802728] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11260.602466] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11269.639356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11270.263754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11277.329763] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11279.495963] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [11287.976050] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11305.440372] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11305.453226] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11305.471087] LustreError: Skipped 1 previous similar message [11305.480446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11305.495907] Lustre: Skipped 1 previous similar message [11308.783598] Lustre: server umount lustre-OST0000 complete [11315.694270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11315.698629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11315.708482] Lustre: Skipped 2 previous similar messages [11315.727928] Lustre: Skipped 1 previous similar message [11320.360741] Lustre: server umount lustre-MDT0000 complete [11320.803391] LustreError: 279021:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11320.826868] LustreError: 279021:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11324.632538] LustreError: 279001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785788251 with bad export cookie 5743851216931094551 [11324.656383] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11325.321797] Lustre: server umount lustre-MDT0001 complete [11339.771278] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [11343.376373] Key type lgssc unregistered [11343.841440] LNet: 282743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11343.848390] LNetError: 282743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11343.875388] LNet: Removed LNI 192.168.204.107@tcp [11345.123320] Key type .llcrypt unregistered [11345.124748] Key type ._llcrypt unregistered [11360.232949] Key type ._llcrypt registered [11360.239961] Key type .llcrypt registered [11360.380886] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [11383.816923] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [11403.659958] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [11405.004038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11405.239519] alg: No test for adler32 (adler32-zlib) [11406.519091] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [11407.016642] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [11408.831187] Key type lgssc registered [11410.435069] Lustre: Echo OBD driver; http://www.lustre.org/ [11452.510936] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11452.539743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11453.953936] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11453.999165] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11454.131266] Lustre: lustre-MDT0000: new disk, initializing [11454.299225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11454.407323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11461.649854] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11482.609635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11483.114115] Lustre: 286299:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11483.359900] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11483.377680] Lustre: Skipped 1 previous similar message [11483.758522] Lustre: lustre-MDT0001: new disk, initializing [11484.023574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11484.133279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11484.170472] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11494.353373] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11508.669541] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11515.879292] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11523.947531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11524.390555] Lustre: lustre-OST0000: new disk, initializing [11524.396959] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11524.406216] Lustre: 288272:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11524.596674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11530.889575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11530.904439] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11531.081664] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11532.518919] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11543.913886] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11550.559782] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11551.003200] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11556.239020] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11556.474276] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11561.960980] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11561.963859] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11562.006743] LustreError: Skipped 1 previous similar message [11562.019883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11562.027551] Lustre: Skipped 1 previous similar message [11562.062752] Lustre: Skipped 1 previous similar message [11567.076093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11567.105901] Lustre: Skipped 1 previous similar message [11572.195741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11572.202419] Lustre: Skipped 1 previous similar message [11573.218731] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11573.789975] Lustre: server umount lustre-OST0000 complete [11581.408681] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11581.427407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11581.480123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11582.435062] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11586.500694] Lustre: server umount lustre-MDT0000 complete [11592.242172] LustreError: 286291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785788518 with bad export cookie 13082889017970751777 [11592.255930] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11592.260292] LustreError: 286291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11593.136423] Lustre: server umount lustre-MDT0001 complete [11605.055779] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 16:22:08 (1785788528) [11615.212078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11615.892639] LustreError: 290691:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11615.914533] LustreError: 290691:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11616.010886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11620.924360] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11621.349712] LustreError: 290692:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11626.465335] LustreError: 290691:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11631.587892] LustreError: 290692:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11633.663706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11634.150109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11641.156783] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11654.221251] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11661.667371] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11671.134928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11671.628167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11681.211605] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11689.742801] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11707.069280] Lustre: server umount lustre-OST0000 complete [11716.068578] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11716.078450] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11716.101715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11716.106782] Lustre: Skipped 2 previous similar messages [11722.719837] Lustre: 284521:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785788633/real 1785788633] req@000000009d4456a0 x1872534787150464/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785788649 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11722.785490] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11722.810903] Lustre: Skipped 1 previous similar message [11722.861719] Lustre: server umount lustre-MDT0000 complete [11724.767429] Lustre: 284522:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785788635/real 1785788635] req@00000000b14f25d6 x1872534787150720/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1785788651 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11727.841391] LustreError: 292170:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11727.883519] LustreError: 292170:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11728.747980] LustreError: 290673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785788655 with bad export cookie 13082889017970752652 [11728.755097] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11728.761785] LustreError: 290673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11729.653607] Lustre: server umount lustre-MDT0001 complete [11747.216542] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [11750.435339] Key type lgssc unregistered [11750.856687] LNet: 294268:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11750.871432] LNetError: 294268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11751.910967] LNet: Removed LNI 192.168.204.107@tcp [11753.243909] Key type .llcrypt unregistered [11753.247084] Key type ._llcrypt unregistered [11788.654524] Key type ._llcrypt registered [11788.665529] Key type .llcrypt registered [11788.963820] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [11815.456493] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [11817.844854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11817.875305] alg: No test for adler32 (adler32-zlib) [11819.212325] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [11819.432399] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [11821.183368] Key type lgssc registered [11822.123814] Lustre: Echo OBD driver; http://www.lustre.org/ [11883.027818] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11883.058132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11884.421177] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11884.487110] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11884.569231] Lustre: lustre-MDT0000: new disk, initializing [11884.668160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11884.681424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11890.169910] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11905.148325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11905.280236] Lustre: 298099:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11905.373826] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11905.384853] Lustre: Skipped 1 previous similar message [11905.548904] Lustre: lustre-MDT0001: new disk, initializing [11905.648681] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11905.722041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11905.744456] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11912.567336] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11925.436272] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11933.436909] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11942.926495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11943.472466] Lustre: lustre-OST2710: new disk, initializing [11943.483355] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [11943.494074] Lustre: 300069:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11943.662965] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [11951.211797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [11951.231213] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [11951.386856] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [11951.601357] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11965.874244] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11976.077681] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11976.310910] Lustre: lustre-OST03e8: new disk, initializing [11976.314342] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [11976.320325] Lustre: 301134:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11976.432480] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [11984.116649] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11985.514520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [11985.523956] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [11985.637988] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [11994.068459] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12003.220658] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [12003.393957] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12009.368900] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [12009.654813] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12014.936413] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [12015.108255] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12020.047714] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [12020.249625] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12028.387815] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12028.393345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12028.410772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12031.973352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12031.985304] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12032.009021] Lustre: Skipped 1 previous similar message [12033.910640] Lustre: server umount lustre-MDT0000 complete [12037.865599] LustreError: 298089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785788964 with bad export cookie 14117223886133218929 [12037.871523] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12037.878595] LustreError: 298089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12038.390086] Lustre: server umount lustre-MDT0001 complete [12054.141349] Lustre: server umount lustre-OST2710 complete [12057.567925] Lustre: 295894:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785788968/real 1785788968] req@00000000c70ab8ed x1872535219154688/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785788984 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12057.609119] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12057.628903] Lustre: Skipped 1 previous similar message [12059.820950] Lustre: server umount lustre-OST03e8 complete [12086.100638] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [12104.466714] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [12149.940656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12150.303890] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12150.345654] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12150.499449] Lustre: lustre-MDT0000: new disk, initializing [12150.628362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12150.647474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12157.002507] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12172.666654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12172.850457] Lustre: 306408:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12172.930267] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12172.940158] Lustre: Skipped 1 previous similar message [12173.115279] Lustre: lustre-MDT0001: new disk, initializing [12173.250564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12173.308899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12173.322795] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12178.563983] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12189.792474] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12197.209520] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12206.434637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12206.799751] Lustre: lustre-OST0000: new disk, initializing [12206.802782] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12206.807789] Lustre: 308381:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12206.877852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12214.407593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12214.432799] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12214.659718] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12214.693809] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12226.788841] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12232.884397] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12233.126182] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12238.948416] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12239.121350] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12245.509111] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12245.510896] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12245.523043] LustreError: Skipped 1 previous similar message [12245.525256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12245.525262] Lustre: Skipped 3 previous similar messages [12245.569538] Lustre: Skipped 1 previous similar message [12247.759795] Lustre: server umount lustre-OST0000 complete [12255.211238] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12255.220145] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12255.246162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12255.257833] Lustre: Skipped 1 previous similar message [12258.927598] Lustre: server umount lustre-MDT0000 complete [12260.839575] LustreError: 306417:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12260.855329] LustreError: 306417:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [12263.394817] LustreError: 306401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785789189 with bad export cookie 14117223886133220623 [12263.395706] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12263.407677] LustreError: 306401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [12263.915147] Lustre: server umount lustre-MDT0001 complete [12275.221773] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 16:33:19 (1785789199) [12277.141149] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [12279.407064] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 16:33:23 (1785789203) [12284.387386] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [12317.374294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12317.842516] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12317.913264] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12318.261416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12324.488701] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12335.823684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12335.928354] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12336.014053] Lustre: 312242:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12341.589206] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12364.731586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12364.964434] Lustre: Denying initial registration attempt from nid 192.168.204.107@tcp, specified as failover [12364.980844] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [12364.993716] LustreError: 313697:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [12365.002347] LustreError: 313697:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [12365.004818] LustreError: 313697:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [12365.047211] Lustre: server umount lustre-OST0000 complete [12365.050698] LustreError: 313697:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [12376.034476] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12376.051970] Lustre: Skipped 1 previous similar message [12376.061039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12376.073658] Lustre: Skipped 1 previous similar message [12386.272467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12386.281026] Lustre: Skipped 3 previous similar messages [12388.319125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12388.722676] Lustre: server umount lustre-MDT0000 complete [12391.392892] LustreError: 312254:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12391.419245] LustreError: 312254:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12394.176650] LustreError: 312234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785789320 with bad export cookie 14117223886133221505 [12394.190456] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12394.198141] LustreError: 312234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12394.580723] Lustre: server umount lustre-MDT0001 complete [12409.250958] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [12414.731963] Key type lgssc unregistered [12415.323283] LNet: 314818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12415.350436] LNetError: 314818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12416.431556] LNet: Removed LNI 192.168.204.107@tcp [12417.386117] Key type .llcrypt unregistered [12417.389170] Key type ._llcrypt unregistered [12431.834360] Key type ._llcrypt registered [12431.839272] Key type .llcrypt registered [12432.034895] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [12433.684321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12433.718154] alg: No test for adler32 (adler32-zlib) [12434.763690] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [12434.961454] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [12436.663157] Key type lgssc registered [12437.770973] Lustre: Echo OBD driver; http://www.lustre.org/ [12495.371226] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [12506.675621] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12506.722691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12507.995722] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12508.026650] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12508.225319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12513.901838] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12525.865119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12526.127334] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12526.204852] Lustre: 317595:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12526.709037] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12534.502884] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12543.852903] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12551.618546] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12575.542693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12575.785495] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12576.073911] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12586.359808] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12596.267424] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12602.853196] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12602.859864] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12602.867241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12602.879716] Lustre: Skipped 1 previous similar message [12605.636035] Lustre: server umount lustre-OST0000 complete [12613.098108] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12613.110603] Lustre: Skipped 2 previous similar messages [12613.112473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12613.127450] Lustre: Skipped 1 previous similar message [12616.693491] Lustre: server umount lustre-MDT0000 complete [12621.569149] LustreError: 317587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785789548 with bad export cookie 14569225523786255239 [12621.578195] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12621.582926] LustreError: 317587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12622.206853] Lustre: server umount lustre-MDT0001 complete [12635.540782] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [12639.323939] Key type lgssc unregistered [12639.718528] LNet: 321170:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12639.721759] LNetError: 321170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12639.756129] LNet: Removed LNI 192.168.204.107@tcp [12640.938325] Key type .llcrypt unregistered [12640.940578] Key type ._llcrypt unregistered [12671.528147] Key type ._llcrypt registered [12671.529604] Key type .llcrypt registered [12671.638706] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [12672.758074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12672.995139] alg: No test for adler32 (adler32-zlib) [12674.157531] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [12674.339766] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [12675.967391] Key type lgssc registered [12677.491705] Lustre: Echo OBD driver; http://www.lustre.org/ [12687.793493] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12687.839672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12689.423373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12696.605802] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12698.605222] LustreError: 322589:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12698.633047] LustreError: 322589:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12703.730886] LustreError: 322590:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12708.724554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12708.832582] LustreError: 322589:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12709.667734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12715.874203] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12727.889402] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12737.725351] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12748.930301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12749.485340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12757.565820] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12768.439392] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12779.489203] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12779.500270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12779.504760] Lustre: Skipped 1 previous similar message [12779.516062] Lustre: Skipped 2 previous similar messages [12780.797174] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.7@tcp (stopping) [12784.275989] Lustre: server umount lustre-MDT0000 complete [12784.615195] LustreError: 322584:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12784.633831] LustreError: 322584:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12788.870264] LustreError: 322569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785789715 with bad export cookie 1017636247905682305 [12788.883276] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12789.728917] LustreError: 322585:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12789.729085] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12789.757930] LustreError: 322585:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12789.778795] Lustre: Skipped 1 previous similar message [12789.787804] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12793.760936] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12793.767451] Lustre: Skipped 1 previous similar message [12795.671988] Lustre: server umount lustre-MDT0001 complete [12804.107638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [12817.794146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12833.825200] LustreError: 321898:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000014f5bc7b x1872536115645952/t0(0) o250->MGC192.168.204.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12834.359833] LustreError: 324423:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12834.386517] LustreError: 324423:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12834.525391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12834.635444] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [12840.627643] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12849.135830] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12852.158975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12852.625193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12857.847891] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12857.983577] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12867.499886] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12869.900363] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12876.543643] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12878.903743] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [12888.551962] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12888.555164] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12888.556086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12888.614415] Lustre: Skipped 1 previous similar message [12898.799440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12898.815219] Lustre: Skipped 3 previous similar messages [12901.343667] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12901.805349] Lustre: server umount lustre-OST0000 complete [12909.024814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12919.269799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12919.283725] Lustre: Skipped 6 previous similar messages [12922.335117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12922.686529] Lustre: server umount lustre-MDT0000 complete [12924.384749] LustreError: 326193:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12924.403663] LustreError: 326193:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [12927.203320] LustreError: 322569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785789853 with bad export cookie 1017636247905689452 [12927.207714] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12927.215402] LustreError: 322569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [12927.718526] Lustre: server umount lustre-MDT0001 complete [12943.695992] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [12947.772635] Key type lgssc unregistered [12948.250300] LNet: 328866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12948.262447] LNetError: 328866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12949.285669] LNet: Removed LNI 192.168.204.107@tcp [12950.257327] Key type .llcrypt unregistered [12950.263329] Key type ._llcrypt unregistered [12978.640683] Key type ._llcrypt registered [12978.643364] Key type .llcrypt registered [12978.767750] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [12980.247361] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12980.273869] alg: No test for adler32 (adler32-zlib) [12981.386584] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [12981.602849] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [12983.271186] Key type lgssc registered [12984.263729] Lustre: Echo OBD driver; http://www.lustre.org/ [12992.774469] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12992.808887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12994.231183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13000.423673] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [13003.757675] LustreError: 330283:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13003.792657] LustreError: 330283:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13008.866348] LustreError: 330284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13010.088284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13010.658386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13015.504868] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [13023.337721] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13029.036812] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13032.920912] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13032.945651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13036.009458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13036.014876] Lustre: Skipped 2 previous similar messages [13041.121446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13046.244763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13046.255919] Lustre: Skipped 2 previous similar messages [13047.263121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13047.421263] Lustre: server umount lustre-MDT0000 complete [13051.364621] LustreError: 331012:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13051.391584] LustreError: 331012:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13052.603171] LustreError: 330265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785789979 with bad export cookie 12089236106558880017 [13052.619908] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13053.131472] Lustre: server umount lustre-MDT0001 complete [13072.089894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13072.577911] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [13072.650693] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13072.856087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13078.763509] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [13089.709461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13089.881123] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13089.982819] Lustre: 332871:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13090.509825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13096.438473] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [13105.216369] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13111.719654] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13123.569397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13123.853166] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [13123.870252] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [13123.880264] LustreError: 334757:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [13123.887895] LustreError: 334757:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [13123.895259] LustreError: 334757:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [13123.978442] Lustre: server umount lustre-OST0000 complete [13123.987231] LustreError: 334757:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [13136.575077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13136.844369] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [13137.212632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13143.665353] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [13144.266074] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [13153.468549] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13162.782204] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13162.891942] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [13162.992667] Lustre: lustre-OST0001: new disk, initializing [13162.995677] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [13162.999691] Lustre: 336301:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13163.078247] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13169.917074] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [13170.815316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13170.827346] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13170.936711] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13178.510452] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13186.528388] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13186.528911] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13186.545239] LustreError: Skipped 1 previous similar message [13186.558779] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [13186.572979] Lustre: Skipped 2 previous similar messages [13186.604563] Lustre: Skipped 1 previous similar message [13196.256093] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13196.482997] Lustre: server umount lustre-OST0001 complete [13196.777995] LustreError: 336559:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13196.798881] LustreError: 336559:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13201.894057] LustreError: 335225:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13201.895874] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13201.917463] LustreError: 335225:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13201.918209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13201.918212] Lustre: Skipped 2 previous similar messages [13201.959480] Lustre: Skipped 1 previous similar message [13212.147340] LustreError: 335226:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13212.171611] LustreError: 335226:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [13215.199283] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13215.613867] Lustre: server umount lustre-OST0000 complete [13222.373064] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13222.374943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13222.382925] Lustre: Skipped 1 previous similar message [13222.398430] Lustre: Skipped 6 previous similar messages [13226.461019] Lustre: server umount lustre-MDT0000 complete [13230.671425] LustreError: 332862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785790157 with bad export cookie 12089236106558880563 [13230.681113] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13230.683377] LustreError: 332862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [13231.248569] Lustre: server umount lustre-MDT0001 complete [13245.073700] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [13248.866697] Key type lgssc unregistered [13249.272945] LNet: 338262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13249.285192] LNetError: 338262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13249.308703] LNet: Removed LNI 192.168.204.107@tcp [13250.322251] Key type .llcrypt unregistered [13250.325602] Key type ._llcrypt unregistered [13329.231339] Key type ._llcrypt registered [13329.233757] Key type .llcrypt registered [13329.358301] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [13358.966745] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [13362.724731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13362.856198] alg: No test for adler32 (adler32-zlib) [13364.696121] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [13365.086063] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [13366.840131] Key type lgssc registered [13370.906983] Lustre: Echo OBD driver; http://www.lustre.org/ [13425.131862] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13425.170335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13426.742768] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13426.814577] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13427.057920] Lustre: lustre-MDT0000: new disk, initializing [13427.304215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13427.379544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13436.556184] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [13454.664638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13454.782101] Lustre: 343207:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13454.860367] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13454.863509] Lustre: Skipped 1 previous similar message [13454.945529] Lustre: lustre-MDT0001: new disk, initializing [13455.064636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13455.119223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13455.131670] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13461.233257] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [13476.983559] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13488.288793] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13499.810752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13500.408642] Lustre: lustre-OST0000: new disk, initializing [13500.410785] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13500.418815] Lustre: 345179:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13500.583803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13506.163327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13506.177643] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13506.336150] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13508.942268] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [13524.433654] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13533.680921] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13533.994465] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13539.116488] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13539.245889] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13542.378712] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13542.380739] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13542.386359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13542.410618] Lustre: Skipped 1 previous similar message [13547.302280] Lustre: server umount lustre-OST0000 complete [13552.614938] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13552.620382] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13552.635569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13552.644078] Lustre: Skipped 1 previous similar message [13552.658949] Lustre: Skipped 2 previous similar messages [13557.733849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13557.742461] Lustre: Skipped 1 previous similar message [13559.147927] Lustre: server umount lustre-MDT0000 complete [13564.031130] LustreError: 343199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785790490 with bad export cookie 9089421752503076925 [13564.032556] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13564.054432] LustreError: 343199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13565.358364] Lustre: server umount lustre-MDT0001 complete [13576.951989] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 16:55:01 (1785790501) [13587.245789] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 16:55:11 (1785790511) [13598.858859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13599.189272] LustreError: 347938:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13599.209516] LustreError: 347938:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [13599.279861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13604.324471] LustreError: 347939:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13605.365721] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [13609.442818] LustreError: 347938:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13614.569802] LustreError: 347939:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13614.714899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13614.988978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13619.644646] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [13628.336687] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13635.822454] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13649.080337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13650.182930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13665.203630] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [13679.550935] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13690.059981] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [13692.910041] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [13694.829386] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [13697.345686] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [13699.912815] Lustre: Failing over lustre-MDT0000 [13700.448120] Lustre: server umount lustre-MDT0000 complete [13700.584883] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13700.599676] LustreError: 347939:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13700.605978] Lustre: Skipped 2 previous similar messages [13700.636086] LustreError: 347939:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13710.826483] LustreError: 348947:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13710.859486] LustreError: 348947:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [13716.967102] Lustre: 341427:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785790627/real 1785790627] req@000000000af47c02 x1872536840258816/t0(0) o400->MGC192.168.204.107@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785790643 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13716.999172] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13722.476589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13727.204721] LustreError: 347933:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13727.255580] LustreError: 347933:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [13727.645906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13727.734064] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [13728.983171] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [13732.374080] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [13732.843679] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [13732.890849] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [13732.969397] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13743.452266] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13745.949460] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13750.462186] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [13752.775093] Lustre: Failing over lustre-MDT0000 [13753.313556] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13753.319817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13753.375946] Lustre: server umount lustre-MDT0000 complete [13759.695502] LustreError: 347933:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13759.716673] LustreError: 347933:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [13768.656561] Lustre: 341427:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785790679/real 1785790679] req@000000006701c1f1 x1872536840280064/t0(0) o400->MGC192.168.204.107@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785790695 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13768.697538] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13775.182087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13779.390387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13779.453979] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [13781.224748] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [13784.573635] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [13784.592794] Lustre: Skipped 2 previous similar messages [13784.658072] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [13784.738200] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13786.123400] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [13798.392232] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13800.616057] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13810.154630] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13810.179659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13810.191130] Lustre: Skipped 4 previous similar messages [13810.215039] Lustre: Skipped 2 previous similar messages [13814.944681] Lustre: server umount lustre-MDT0000 complete [13821.399646] LustreError: 347920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785790747 with bad export cookie 9089421752503079634 [13821.408659] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13821.419406] LustreError: 347920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13822.658396] Lustre: server umount lustre-MDT0001 complete [13838.451376] Lustre: server umount lustre-OST0000 complete [13859.035602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13859.429660] LustreError: 354287:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13859.446683] LustreError: 354287:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 22 previous similar messages [13859.492575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13865.481684] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [13876.516330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13883.016327] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [13893.059412] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13900.328699] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13909.536457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13918.206443] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [13929.013941] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13929.599292] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13938.664042] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13938.677683] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13938.699589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13942.232563] Lustre: server umount lustre-OST0000 complete [13948.896145] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13948.910815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13948.920142] Lustre: Skipped 2 previous similar messages [13954.471221] Lustre: server umount lustre-MDT0000 complete [13962.427256] LustreError: 354267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785790888 with bad export cookie 9089421752503080383 [13962.437553] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13962.439497] LustreError: 354267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13962.811672] Lustre: server umount lustre-MDT0001 complete [13982.081326] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [13985.225554] Key type lgssc unregistered [13985.670625] LNet: 357855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13985.691133] LNetError: 357855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13986.724550] LNet: Removed LNI 192.168.204.107@tcp [13988.035404] Key type .llcrypt unregistered [13988.040150] Key type ._llcrypt unregistered [14018.817407] Key type ._llcrypt registered [14018.826345] Key type .llcrypt registered [14019.059794] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [14021.493366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14021.552431] alg: No test for adler32 (adler32-zlib) [14022.985635] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [14023.597587] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [14025.383275] Key type lgssc registered [14027.966914] Lustre: Echo OBD driver; http://www.lustre.org/ [14045.697294] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14045.733290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14047.552896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14054.387225] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [14057.954484] LustreError: 359373:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14057.988832] LustreError: 359373:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14063.080683] LustreError: 359374:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14066.967874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14067.668832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14075.713818] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [14087.198654] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14096.106323] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14104.515469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14104.961026] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14113.299443] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [14118.350323] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [14122.208347] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14138.854467] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14138.873041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14138.881299] Lustre: Skipped 2 previous similar messages [14142.704394] Lustre: server umount lustre-MDT0000 complete [14143.971769] LustreError: 359373:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14143.993556] LustreError: 359373:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14147.213663] LustreError: 359354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785791073 with bad export cookie 1431003177233871784 [14147.225312] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14147.235245] LustreError: 359354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14147.987755] Lustre: server umount lustre-MDT0001 complete [14163.274872] Lustre: server umount lustre-OST0000 complete [14179.786091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14180.411980] LustreError: 363113:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14180.521547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14185.953636] LustreError: 363114:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14187.712957] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [14195.168343] LustreError: 363113:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14195.185371] LustreError: 363113:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14197.724104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14198.168625] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14204.141952] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [14213.974883] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14221.774609] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14231.602579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14232.335100] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14241.107255] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [14249.745113] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [14256.785797] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14280.160560] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14280.169516] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14280.180507] Lustre: Skipped 2 previous similar messages [14280.183879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14283.337340] Lustre: server umount lustre-MDT0000 complete [14285.796343] LustreError: 363841:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14285.828348] LustreError: 363841:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14288.247542] LustreError: 363093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785791214 with bad export cookie 1431003177233872897 [14288.261117] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14288.706712] Lustre: server umount lustre-MDT0001 complete [14304.655760] Lustre: server umount lustre-OST0000 complete [14328.554734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14329.478431] LustreError: 367026:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14329.659470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14336.546581] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [14347.586657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14353.254314] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [14361.831986] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14369.595138] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14377.699180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14378.208364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14378.213271] Lustre: Skipped 1 previous similar message [14386.331977] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [14397.027743] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14399.781789] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [14410.211225] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14410.214254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14410.237571] Lustre: Skipped 3 previous similar messages [14410.257555] Lustre: Skipped 2 previous similar messages [14414.611420] Lustre: server umount lustre-OST0000 complete [14419.935870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14419.943816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14419.964445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14419.975016] Lustre: Skipped 1 previous similar message [14425.568196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14425.576128] Lustre: Skipped 3 previous similar messages [14434.271314] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14434.804577] LustreError: 368177:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14434.843674] LustreError: 368177:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [14434.853535] Lustre: server umount lustre-MDT0000 complete [14440.882527] LustreError: 367007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785791367 with bad export cookie 1431003177233873919 [14440.892216] LustreError: 367007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14440.898211] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14441.949560] Lustre: server umount lustre-MDT0001 complete [14461.069206] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [14464.098266] Key type lgssc unregistered [14464.393589] LNet: 370587:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14464.399804] LNetError: 370587:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14464.425094] LNet: Removed LNI 192.168.204.107@tcp [14466.126270] Key type .llcrypt unregistered [14466.127913] Key type ._llcrypt unregistered [14499.287792] Key type ._llcrypt registered [14499.288996] Key type .llcrypt registered [14499.405371] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [14500.719430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14500.843846] alg: No test for adler32 (adler32-zlib) [14501.959562] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [14502.218527] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [14503.874614] Key type lgssc registered [14505.667628] Lustre: Echo OBD driver; http://www.lustre.org/ [14516.306960] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14516.331399] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14516.342647] LustreError: 372030:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [14516.386586] LustreError: 372030:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [14516.393645] LustreError: 372030:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [14516.405470] LustreError: 372030:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [14516.418256] LustreError: 372030:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [14526.561539] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [14526.571784] LustreError: 372428:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [14526.602691] LustreError: 372428:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [14526.610340] LustreError: 372428:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [14526.618757] LustreError: 372428:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [14526.628435] LustreError: 372428:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [14557.929234] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [14562.710611] Key type lgssc unregistered [14563.346370] LNet: 373451:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14563.365259] LNetError: 373451:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14563.387601] LNet: Removed LNI 192.168.204.107@tcp [14565.123924] Key type .llcrypt unregistered [14565.138376] Key type ._llcrypt unregistered [14589.865558] Key type ._llcrypt registered [14589.866928] Key type .llcrypt registered [14590.049024] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [14617.647974] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [14619.600760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14619.839786] alg: No test for adler32 (adler32-zlib) [14621.550474] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [14621.912886] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [14623.751149] Key type lgssc registered [14626.100339] Lustre: Echo OBD driver; http://www.lustre.org/ [14677.176282] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14677.203710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14678.517226] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14678.565801] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14678.752210] Lustre: lustre-MDT0000: new disk, initializing [14678.921691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14678.970360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14684.023294] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [14698.582289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14698.745699] Lustre: 376613:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14698.833860] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14698.842152] Lustre: Skipped 1 previous similar message [14698.976514] Lustre: lustre-MDT0001: new disk, initializing [14699.121192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14699.170264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14699.189848] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14705.069538] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [14717.048562] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14724.891792] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14734.337287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14734.936420] Lustre: lustre-OST0000: new disk, initializing [14734.947958] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14734.963558] Lustre: 378583:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14735.224296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14741.111633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14741.138647] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14741.358947] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14743.871502] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [14755.642609] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14763.667545] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14763.960436] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14769.594281] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14769.899208] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14777.318565] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14777.320036] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14777.352500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14777.365483] Lustre: Skipped 1 previous similar message [14779.017965] Lustre: server umount lustre-OST0000 complete [14786.016275] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14786.034276] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14786.065303] Lustre: Skipped 1 previous similar message [14786.080738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14787.553752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14787.560155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14787.593161] Lustre: Skipped 1 previous similar message [14790.297933] Lustre: server umount lustre-MDT0000 complete [14795.060514] LustreError: 376604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785791721 with bad export cookie 1991150856968175151 [14795.068960] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14795.083332] LustreError: 376604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14795.716848] Lustre: server umount lustre-MDT0001 complete [14807.198188] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [14809.204684] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 17:15:33 (1785791733) [14817.981356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14818.913270] LustreError: 381047:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14818.939133] LustreError: 381047:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [14819.050305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14824.420571] LustreError: 381048:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14824.440300] LustreError: 381048:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14825.510751] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [14828.517633] LustreError: 381047:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14833.636055] LustreError: 381048:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14835.572849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14836.040858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14842.599936] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [14852.300552] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14860.704837] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14875.668543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14876.619213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14888.027659] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [14898.288950] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14908.676649] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14909.266206] Lustre: lustre-OST0001: new disk, initializing [14909.289386] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [14909.347693] Lustre: 383887:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14909.685824] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14917.857955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14917.877881] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14918.029562] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14921.504520] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [14941.827517] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14948.833631] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14948.842850] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14948.855539] Lustre: Skipped 1 previous similar message [14948.863925] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [14953.369744] Lustre: server umount lustre-OST0001 complete [14953.963625] LustreError: 382871:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14953.995470] LustreError: 382871:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14964.204938] LustreError: 382871:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14964.241431] LustreError: 382871:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [14978.018082] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14978.036947] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14978.068175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14978.081174] Lustre: Skipped 2 previous similar messages [14984.679718] LustreError: 382871:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14984.704249] LustreError: 382871:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [14989.809570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14989.829270] Lustre: Skipped 5 previous similar messages [14991.327282] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14991.829073] Lustre: server umount lustre-OST0000 complete [15000.038107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15000.054598] Lustre: Skipped 1 previous similar message [15003.966343] Lustre: server umount lustre-MDT0000 complete [15009.200696] LustreError: 383886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785791935 with bad export cookie 1991150856968176033 [15009.221336] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15009.224093] LustreError: 383886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15016.029734] Lustre: server umount lustre-MDT0001 complete [15033.892750] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [15038.268254] Key type lgssc unregistered [15038.761181] LNet: 385898:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15038.777064] LNetError: 385898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15038.814101] LNet: Removed LNI 192.168.204.107@tcp [15040.373153] Key type .llcrypt unregistered [15040.388328] Key type ._llcrypt unregistered [15109.154185] Key type ._llcrypt registered [15109.157970] Key type .llcrypt registered [15109.263309] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [15110.884195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15110.941264] alg: No test for adler32 (adler32-zlib) [15112.298832] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [15112.594162] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [15114.375305] Key type lgssc registered [15115.812130] Lustre: Echo OBD driver; http://www.lustre.org/ [15125.327081] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15125.352513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15126.710467] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [15126.754969] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15127.003457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15134.015098] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [15145.976270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15146.032540] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [15146.083156] Lustre: 388702:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15146.331229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15153.544869] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [15167.543886] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15174.825152] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15179.744490] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15179.746246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15179.764740] Lustre: Skipped 1 previous similar message [15179.791832] Lustre: Skipped 1 previous similar message [15184.654277] Lustre: server umount lustre-MDT0000 complete [15189.987966] LustreError: 388714:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15190.036336] LustreError: 388714:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [15190.437244] LustreError: 388696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785792116 with bad export cookie 9287405900890192597 [15190.441672] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15190.453972] LustreError: 388696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15191.417738] Lustre: server umount lustre-MDT0001 complete [15208.953724] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 17:22:11 (1785792131) [15220.507751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15221.140236] LustreError: 391222:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15221.321506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15226.339268] LustreError: 391223:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15227.048116] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [15231.476817] LustreError: 391222:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15236.576164] LustreError: 391223:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15242.309508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15242.588454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15250.491843] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [15261.081610] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15268.243968] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15276.791300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15277.021505] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [15277.247382] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15286.203088] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [15298.911930] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15309.632910] Lustre: Permanently deactivating lustre-OST0000 [15309.661859] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [15316.003805] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [15318.969488] LustreError: 394166:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [15318.983176] LustreError: 394166:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [15318.991405] LustreError: 394166:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [15324.445686] LustreError: 394214:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [15324.455935] LustreError: 394214:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [15329.716497] Lustre: server umount lustre-OST0000 complete [15333.499579] LustreError: 394464:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [15333.515576] LustreError: 394464:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [15339.498697] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15339.553098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15339.558337] Lustre: Skipped 1 previous similar message [15344.610395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15344.619182] Lustre: Skipped 1 previous similar message [15346.752696] Lustre: server umount lustre-MDT0000 complete [15349.729478] LustreError: 391217:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15349.747248] LustreError: 391217:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15352.258443] LustreError: 391204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785792278 with bad export cookie 9287405900890193178 [15352.284991] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15361.757663] Lustre: server umount lustre-MDT0001 complete [15375.794189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15381.463785] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [15385.232495] LustreError: 395626:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [15393.516037] Lustre: 395966:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.107@tcp [15396.234736] Lustre: server umount MGS complete [15411.659705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15412.446314] LustreError: 396691:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15412.475449] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [15412.484528] Lustre: Skipped 1 previous similar message [15412.597267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15418.450796] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [15430.918759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15431.497139] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [15431.693686] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15438.850116] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [15451.059846] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15461.109366] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15464.979321] Lustre: Permanently reactivating lustre-OST0000 [15465.021832] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [15465.032099] Lustre: Skipped 2 previous similar messages [15493.599636] LustreError: 398436:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [15504.190030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15504.723077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15513.274402] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [15522.805155] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15533.540792] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15533.550673] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15533.569435] Lustre: Skipped 1 previous similar message [15533.572833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15534.573928] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15538.186440] Lustre: server umount lustre-OST0000 complete [15544.288556] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15544.296789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15544.318560] Lustre: Skipped 1 previous similar message [15544.327449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15544.330106] Lustre: Skipped 2 previous similar messages [15549.771531] Lustre: server umount lustre-MDT0000 complete [15549.925733] LustreError: 397420:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15549.949053] LustreError: 397420:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [15553.464481] LustreError: 396673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785792480 with bad export cookie 9287405900890194228 [15553.467127] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15553.473791] LustreError: 396673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [15553.910330] Lustre: server umount lustre-MDT0001 complete [15568.558549] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [15572.183471] Key type lgssc unregistered [15572.546581] LNet: 400965:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15572.550990] LNetError: 400965:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15572.581221] LNet: Removed LNI 192.168.204.107@tcp [15573.714453] Key type .llcrypt unregistered [15573.718433] Key type ._llcrypt unregistered [15598.050456] Key type ._llcrypt registered [15598.054768] Key type .llcrypt registered [15598.273871] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [15623.506566] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [15625.685503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15625.748838] alg: No test for adler32 (adler32-zlib) [15627.180810] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [15627.711507] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [15629.543644] Key type lgssc registered [15631.798667] Lustre: Echo OBD driver; http://www.lustre.org/ [15684.362754] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15684.392039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15685.901036] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15685.970187] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15686.195829] Lustre: lustre-MDT0000: new disk, initializing [15686.353274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15686.384903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15691.957754] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [15705.485716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15705.610406] Lustre: 404131:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15705.650649] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15705.653232] Lustre: Skipped 1 previous similar message [15705.706342] Lustre: lustre-MDT0001: new disk, initializing [15705.779179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15705.850624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15705.878199] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15712.240287] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [15726.097976] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15737.014437] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15746.570136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15747.082337] Lustre: lustre-OST0000: new disk, initializing [15747.092047] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15747.110764] Lustre: 406102:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15747.286762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15754.856850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15754.879321] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15755.038260] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15756.637327] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [15769.547907] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15774.864350] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15775.057975] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15779.490301] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15779.630841] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15785.955137] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15785.964900] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15785.965648] LustreError: Skipped 1 previous similar message [15785.971709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15785.980889] Lustre: Skipped 1 previous similar message [15786.011819] Lustre: Skipped 1 previous similar message [15788.038721] Lustre: server umount lustre-OST0000 complete [15796.194546] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15796.198821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15796.211811] Lustre: Skipped 1 previous similar message [15796.225441] Lustre: Skipped 1 previous similar message [15801.316224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15801.323662] Lustre: Skipped 1 previous similar message [15802.386594] Lustre: server umount lustre-MDT0000 complete [15808.577596] LustreError: 406101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785792735 with bad export cookie 209060244929320207 [15808.584241] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15808.601753] LustreError: 406101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15809.802033] Lustre: server umount lustre-MDT0001 complete [15827.645596] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [15832.285426] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 17:32:34 (1785792754) [15852.505753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15854.137514] LustreError: 408567:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15854.188988] LustreError: 408567:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [15854.350260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15859.682823] LustreError: 408568:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15860.493771] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [15863.776344] LustreError: 408567:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15868.906311] LustreError: 408568:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15872.139949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15872.725833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15879.659918] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [15889.233924] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15898.810686] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15909.745854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15910.350347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15920.269934] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [15934.972940] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15938.285397] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [15943.069163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [15953.377862] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15953.397288] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15953.427844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15954.404364] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15955.488559] Lustre: server umount lustre-OST0000 complete [15964.640289] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15964.644097] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15964.690496] Lustre: Skipped 1 previous similar message [15964.706526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15964.706544] Lustre: Skipped 2 previous similar messages [15974.898163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15974.907308] Lustre: Skipped 4 previous similar messages [15975.391425] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15975.986225] Lustre: server umount lustre-MDT0000 complete [15980.003259] LustreError: 408567:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15980.025636] LustreError: 408567:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15981.194578] LustreError: 408547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785792907 with bad export cookie 209060244929321089 [15981.196649] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15981.216037] LustreError: 408547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15982.040160] Lustre: server umount lustre-MDT0001 complete [15998.695367] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [16003.248670] Key type lgssc unregistered [16003.788824] LNet: 412301:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16003.807845] LNetError: 412301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16003.827714] LNet: Removed LNI 192.168.204.107@tcp [16005.390696] Key type .llcrypt unregistered [16005.395162] Key type ._llcrypt unregistered [16041.183324] Key type ._llcrypt registered [16041.187243] Key type .llcrypt registered [16041.308669] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [16041.556588] Key type .llcrypt unregistered [16041.559181] Key type ._llcrypt unregistered [16064.860224] Key type ._llcrypt registered [16064.866911] Key type .llcrypt registered [16065.048972] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [16067.163274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16067.213355] alg: No test for adler32 (adler32-zlib) [16068.679444] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [16069.128209] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [16070.999883] Key type lgssc registered [16073.205359] Lustre: Echo OBD driver; http://www.lustre.org/ [16084.263582] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16084.317806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16086.070969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16093.890857] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [16096.232867] LustreError: 414519:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16096.265731] LustreError: 414519:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16101.352972] LustreError: 414520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16105.834293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16106.468120] LustreError: 414519:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16106.665843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16110.763027] LustreError: 414520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16110.785105] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [16110.801063] LustreError: 414520:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16116.199905] LustreError: 414516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16116.238616] LustreError: 414516:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16116.857314] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [16126.453121] LustreError: 414516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16126.499061] LustreError: 414516:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [16131.616685] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16145.977717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16146.967990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16155.965068] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [16172.515693] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16172.517287] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16172.543052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16172.548450] Lustre: Skipped 1 previous similar message [16172.568723] Lustre: Skipped 1 previous similar message [16176.627616] Lustre: server umount lustre-OST0000 complete [16182.755671] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16182.756182] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16182.766127] LustreError: Skipped 1 previous similar message [16182.772487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16182.801125] Lustre: Skipped 1 previous similar message [16187.876443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16187.889224] Lustre: Skipped 2 previous similar messages [16189.011883] Lustre: server umount lustre-MDT0000 complete [16192.997329] LustreError: 416259:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16193.031674] LustreError: 416259:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [16194.306729] LustreError: 414500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785793120 with bad export cookie 5325033764206813746 [16194.323095] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16194.331157] LustreError: 414500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16195.134919] Lustre: server umount lustre-MDT0001 complete [16210.635444] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [16215.056652] Key type lgssc unregistered [16215.527171] LNet: 417674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16215.537202] LNetError: 417674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16215.557272] LNet: Removed LNI 192.168.204.107@tcp [16216.815096] Key type .llcrypt unregistered [16216.816500] Key type ._llcrypt unregistered [16256.279270] Key type ._llcrypt registered [16256.280396] Key type .llcrypt registered [16256.416924] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [16257.719287] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16257.759094] alg: No test for adler32 (adler32-zlib) [16259.345684] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [16259.797924] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [16261.583769] Key type lgssc registered [16263.585694] Lustre: Echo OBD driver; http://www.lustre.org/ [16275.702673] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16275.740158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16291.423501] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [16296.543905] LustreError: 419073:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.107@tcp: failed processing log, type 4: rc = -110 [16323.295367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16332.016701] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [16342.386025] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16353.219953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16354.589417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16354.719525] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [16362.368275] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [16362.979257] LustreError: 419098:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16363.006736] LustreError: 419098:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [16368.102669] LustreError: 420113:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16368.140351] LustreError: 420113:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16373.246884] LustreError: 419098:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16373.266411] LustreError: 419098:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16373.899321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16374.499799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16381.083825] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [16391.811852] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16399.799196] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16410.085219] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16410.100858] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16410.127973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16410.605780] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16414.958752] Lustre: server umount lustre-OST0000 complete [16425.441301] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16425.454694] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16425.476968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16425.483250] Lustre: Skipped 1 previous similar message [16427.949786] Lustre: server umount lustre-MDT0000 complete [16431.082743] LustreError: 420116:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16431.110358] LustreError: 420116:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16435.411970] LustreError: 419080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785793361 with bad export cookie 13197127991941512731 [16435.446311] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16435.448448] LustreError: 419080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16437.093807] Lustre: server umount lustre-MDT0001 complete [16460.152173] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [16464.478926] Key type lgssc unregistered [16464.903746] LNet: 422667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16464.912576] LNetError: 422667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16464.927711] LNet: Removed LNI 192.168.204.107@tcp [16465.977667] Key type .llcrypt unregistered [16465.982368] Key type ._llcrypt unregistered [16522.418584] Key type ._llcrypt registered [16522.420164] Key type .llcrypt registered [16522.596614] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [16524.570762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16524.689691] alg: No test for adler32 (adler32-zlib) [16526.040504] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [16526.531810] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [16528.407189] Key type lgssc registered [16530.983041] Lustre: Echo OBD driver; http://www.lustre.org/ [16544.006440] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16544.034404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16545.458884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16550.713543] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [16554.990720] LustreError: 424092:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16555.023243] LustreError: 424092:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16560.113260] LustreError: 424093:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16562.533581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16563.052350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16569.345762] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [16580.685588] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16589.187709] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16600.500443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16601.006872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16611.789375] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [16615.412862] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [16623.249566] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16629.744181] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16629.758056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16629.782515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16630.386286] Lustre: server umount lustre-MDT0000 complete [16630.548916] LustreError: 424828:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [16630.754573] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16630.785672] Lustre: Skipped 1 previous similar message [16633.655947] LustreError: 424073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.7@tcp arrived at 1785793560 with bad export cookie 16861454564070963131 [16635.880164] LustreError: 424088:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16635.905246] LustreError: 424088:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [16636.341968] LustreError: 424071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785793562 with bad export cookie 16861454564070962235 [16636.358954] LustreError: 424071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16636.375893] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16638.945567] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16638.961427] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16638.991033] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16640.992706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16641.000567] LustreError: 424089:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16641.035812] LustreError: 424089:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16642.915038] Lustre: server umount lustre-OST0000 complete [16651.238604] LustreError: 424828:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16651.267681] LustreError: 424828:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [16653.064353] Lustre: server umount lustre-MDT0001 complete [16672.349874] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [16676.719741] Key type lgssc unregistered [16676.983398] LNet: 427798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16676.993607] LNetError: 427798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16677.024354] LNet: Removed LNI 192.168.204.107@tcp [16678.160873] Key type .llcrypt unregistered [16678.163382] Key type ._llcrypt unregistered [16714.480701] Key type ._llcrypt registered [16714.487142] Key type .llcrypt registered [16714.604361] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [16716.465893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16716.501097] alg: No test for adler32 (adler32-zlib) [16717.791811] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [16718.181222] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [16719.927181] Key type lgssc registered [16721.330190] Lustre: Echo OBD driver; http://www.lustre.org/ [16733.468234] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16733.511728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16735.493283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16744.487703] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [16744.929335] LustreError: 429216:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16744.961768] LustreError: 429216:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16750.078744] LustreError: 429217:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16754.751237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16755.170301] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [16755.534775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16762.105875] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [16773.486719] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16783.175714] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16793.241842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16793.766362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16802.770787] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [16805.053962] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [16815.095206] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16820.708171] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16820.713838] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [16820.725583] Lustre: Skipped 1 previous similar message [16822.005321] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.7@tcp (stopping) [16822.011421] Lustre: Skipped 1 previous similar message [16823.189818] Lustre: server umount lustre-MDT0001 complete [16825.837985] LustreError: 431253:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16825.863638] LustreError: 431253:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16830.948892] LustreError: 431253:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16830.970572] LustreError: 431253:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16836.065213] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16836.067946] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16836.072341] LustreError: 429216:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16836.139431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16840.070437] Lustre: server umount lustre-OST0000 complete [16846.305105] LustreError: 429217:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16846.341528] LustreError: 429217:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16848.056697] Lustre: server umount lustre-MDT0000 complete [16870.109222] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [16874.520525] Key type lgssc unregistered [16874.809250] LNet: 432906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16874.815492] LNetError: 432906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16874.832857] LNet: Removed LNI 192.168.204.107@tcp [16875.970239] Key type .llcrypt unregistered [16875.973307] Key type ._llcrypt unregistered [16915.395478] Key type ._llcrypt registered [16915.405749] Key type .llcrypt registered [16915.618429] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [16937.898146] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [16939.958768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16940.020655] alg: No test for adler32 (adler32-zlib) [16941.346544] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [16941.748389] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [16943.447426] Key type lgssc registered [16944.684350] Lustre: Echo OBD driver; http://www.lustre.org/ [17003.204435] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [17026.842194] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17026.894403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17028.159637] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [17028.181418] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [17028.289053] Lustre: lustre-MDT0000: new disk, initializing [17028.375693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17028.394711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [17034.342178] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [17057.032501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17057.429358] Lustre: lustre-OST0000: new disk, initializing [17057.439315] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [17057.443793] Lustre: Skipped 1 previous similar message [17057.452366] Lustre: 437971:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17057.560903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17066.134742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [17066.156739] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [17066.209315] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [17066.644387] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [17078.785031] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17099.070229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17099.352567] Lustre: 436951:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [17099.528187] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [17099.698360] Lustre: lustre-MDT0001: new disk, initializing [17100.013366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17100.089825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [17100.117563] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [17106.648660] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [17119.939250] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [17120.334260] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [17127.765919] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [17128.048880] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [17135.334633] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.7@tcp (stopping) [17135.587255] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [17135.593468] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17135.626052] Lustre: Skipped 1 previous similar message [17140.452283] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.7@tcp (stopping) [17140.461537] Lustre: Skipped 2 previous similar messages [17141.706347] Lustre: server umount lustre-MDT0001 complete [17150.955603] LustreError: 436963:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17150.991365] LustreError: 436963:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [17156.071327] LustreError: 436957:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17156.087319] LustreError: 436957:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [17161.209466] LustreError: 438692:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17161.240890] LustreError: 438692:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [17163.750335] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [17163.771227] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17163.801375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17163.809819] Lustre: Skipped 2 previous similar messages [17166.305065] LustreError: 436964:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17166.306973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17170.238839] Lustre: server umount lustre-OST0000 complete [17171.430760] LustreError: 436963:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17182.373801] Lustre: server umount lustre-MDT0000 complete [17201.345161] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [17205.486368] Key type lgssc unregistered [17205.966485] LNet: 441727:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17205.974840] LNetError: 441727:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17206.002946] LNet: Removed LNI 192.168.204.107@tcp [17207.078884] Key type .llcrypt unregistered [17207.080268] Key type ._llcrypt unregistered [17265.606971] Key type ._llcrypt registered [17265.611637] Key type .llcrypt registered [17265.687728] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [17266.097522] Key type .llcrypt unregistered [17266.099145] Key type ._llcrypt unregistered [17289.706221] Key type ._llcrypt registered [17289.710584] Key type .llcrypt registered [17289.924923] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [17292.103423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [17292.181031] alg: No test for adler32 (adler32-zlib) [17293.613953] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [17294.060108] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [17295.880063] Key type lgssc registered [17298.069573] Lustre: Echo OBD driver; http://www.lustre.org/ [17338.701700] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17338.744226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17340.147896] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [17340.214836] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [17340.453312] Lustre: lustre-MDT0000: new disk, initializing [17340.632447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17340.674774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [17345.837881] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [17360.462167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17360.703710] Lustre: 445112:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [17360.834220] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [17360.840690] Lustre: Skipped 1 previous similar message [17361.079395] Lustre: lustre-MDT0001: new disk, initializing [17361.302916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17361.398773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [17361.425314] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [17368.376810] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [17381.418616] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17391.001434] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17401.826080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17402.638385] Lustre: lustre-OST0000: new disk, initializing [17402.644701] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [17402.655538] Lustre: 447081:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17402.803924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17412.355370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [17412.397358] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [17412.509870] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [17414.028389] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [17430.127041] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17443.312460] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17443.313106] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [17443.333801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17443.347120] Lustre: Skipped 2 previous similar messages [17448.849754] Lustre: server umount lustre-MDT0000 complete [17453.538473] LustreError: 445119:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17453.567394] LustreError: 445119:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [17454.159422] LustreError: 445103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785794380 with bad export cookie 2168917412627955006 [17454.166569] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17454.184968] LustreError: 445103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [17454.946024] Lustre: server umount lustre-MDT0001 complete [17470.463952] Lustre: server umount lustre-OST0000 complete [17488.218245] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 18:00:11 (1785794411) [17498.425225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17499.274795] LustreError: 449114:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17499.426913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17504.737357] LustreError: 449115:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17507.259836] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [17508.836745] LustreError: 449114:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17513.959170] LustreError: 449115:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17517.766330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17518.045035] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17524.809922] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [17537.271057] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17545.399272] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17566.571723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17566.676578] Lustre: Found index 0 for lustre-OST0000, updating log [17566.792338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17568.897097] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [17576.914585] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [17587.672429] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17594.862815] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [17594.881582] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17594.898485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17594.907275] Lustre: Skipped 2 previous similar messages [17595.937551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17595.945686] Lustre: Skipped 1 previous similar message [17598.722576] Lustre: server umount lustre-OST0000 complete [17617.384563] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17617.402661] Lustre: Skipped 2 previous similar messages [17617.405215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17617.423271] Lustre: Skipped 1 previous similar message [17621.537326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17621.543321] Lustre: Skipped 1 previous similar message [17623.554895] Lustre: server umount lustre-MDT0000 complete [17627.622250] LustreError: 449111:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17627.654943] LustreError: 449111:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [17629.550363] LustreError: 449095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785794556 with bad export cookie 2168917412627956399 [17629.559960] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17629.571451] LustreError: 449095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [17630.475657] Lustre: server umount lustre-MDT0001 complete [17634.212730] LNet: 1 local NIs in recovery (showing 1): 192.168.204.107@tcp [17645.506736] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 18:02:48 (1785794568) [18298.618212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18300.093316] LustreError: 455548:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18300.305634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [18305.509700] LustreError: 455549:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18309.616751] LustreError: 455548:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18309.693209] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [18320.864495] LustreError: 455549:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18320.915394] LustreError: 455549:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [18329.280051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18330.352389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [18338.169282] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [18350.448749] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [18351.263571] LNet: 1 local NIs in recovery (showing 1): 192.168.204.107@tcp [18358.620334] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [18381.251775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [18381.840575] Lustre: Found index 0 for lustre-OST0000, updating log [18382.776852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [18394.056338] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [18404.536681] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [18407.757206] Lustre: server umount lustre-OST0000 complete [18413.477376] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18413.489949] Lustre: Skipped 1 previous similar message [18413.490032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18413.520893] Lustre: Skipped 3 previous similar messages [18414.564325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18414.570015] Lustre: Skipped 1 previous similar message [18414.751216] LNet: 1 local NIs in recovery (showing 1): 192.168.204.107@tcp [18416.678966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18416.685508] Lustre: Skipped 3 previous similar messages [18418.869266] Lustre: server umount lustre-MDT0000 complete [18419.745678] LustreError: 455549:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18419.793120] LustreError: 455549:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [18423.912238] LustreError: 455529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785795350 with bad export cookie 2168917412627957477 [18423.922454] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [18423.929780] LustreError: 455529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [18424.596879] Lustre: server umount lustre-MDT0001 complete [18461.292673] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [18480.650572] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [18528.887722] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 18:17:32 (1785795452) [18552.041640] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [18555.910923] Key type lgssc unregistered [18556.522476] LNet: 462191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [18556.531344] LNetError: 462191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18642.600655] LNet: Removed LNI 192.168.204.107@tcp [18643.908273] Key type .llcrypt unregistered [18643.913142] Key type ._llcrypt unregistered [18669.465740] Key type ._llcrypt registered [18669.469723] Key type .llcrypt registered [18669.630684] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [18671.033465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [18671.117499] alg: No test for adler32 (adler32-zlib) [18672.413717] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [18672.797115] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [18674.591349] Key type lgssc registered [18676.138901] Lustre: Echo OBD driver; http://www.lustre.org/ [18682.796264] LNet: Added LNI 192.168.204.107@tcp100 [8/256/0/180] [18684.305860] LNet: Added LNI 192.168.204.107@tcp101 [8/256/0/180] [18685.700764] LNet: Added LNI 192.168.204.107@tcp102 [8/256/0/180] [18687.472870] LNet: Added LNI 192.168.204.107@tcp103 [8/256/0/180] [18689.098731] LNet: Added LNI 192.168.204.107@tcp104 [8/256/0/180] [18690.357313] LNet: Added LNI 192.168.204.107@tcp105 [8/256/0/180] [18691.793216] LNet: Added LNI 192.168.204.107@tcp106 [8/256/0/180] [18693.335899] LNet: Added LNI 192.168.204.107@tcp107 [8/256/0/180] [18694.829058] LNet: Added LNI 192.168.204.107@tcp108 [8/256/0/180] [18696.619424] LNet: Added LNI 192.168.204.107@tcp109 [8/256/0/180] [18698.209618] LNet: Added LNI 192.168.204.107@tcp110 [8/256/0/180] [18699.487795] LNet: Added LNI 192.168.204.107@tcp111 [8/256/0/180] [18700.755051] LNet: Added LNI 192.168.204.107@tcp112 [8/256/0/180] [18702.027164] LNet: Added LNI 192.168.204.107@tcp113 [8/256/0/180] [18702.997962] LNet: Added LNI 192.168.204.107@tcp114 [8/256/0/180] [18703.932679] LNet: Added LNI 192.168.204.107@tcp115 [8/256/0/180] [18705.059622] LNet: Added LNI 192.168.204.107@tcp116 [8/256/0/180] [18706.165328] LNet: Added LNI 192.168.204.107@tcp117 [8/256/0/180] [18707.259468] LNet: Added LNI 192.168.204.107@tcp118 [8/256/0/180] [18708.069064] LNet: Added LNI 192.168.204.107@tcp119 [8/256/0/180] [18708.976531] LNet: Added LNI 192.168.204.107@tcp120 [8/256/0/180] [18710.107902] LNet: Added LNI 192.168.204.107@tcp121 [8/256/0/180] [18711.310550] LNet: Added LNI 192.168.204.107@tcp122 [8/256/0/180] [18712.578535] LNet: Added LNI 192.168.204.107@tcp123 [8/256/0/180] [18713.736541] LNet: Added LNI 192.168.204.107@tcp124 [8/256/0/180] [18715.051846] LNet: Added LNI 192.168.204.107@tcp125 [8/256/0/180] [18716.276388] LNet: Added LNI 192.168.204.107@tcp126 [8/256/0/180] [18717.518628] LNet: Added LNI 192.168.204.107@tcp127 [8/256/0/180] [18719.074895] LNet: Added LNI 192.168.204.107@tcp128 [8/256/0/180] [18720.550766] LNet: Added LNI 192.168.204.107@tcp129 [8/256/0/180] [18721.935023] LNet: Added LNI 192.168.204.107@tcp130 [8/256/0/180] [18723.506452] LNet: Added LNI 192.168.204.107@tcp131 [8/256/0/180] [18725.081146] LNet: Added LNI 192.168.204.107@tcp132 [8/256/0/180] [18726.475384] LNet: Added LNI 192.168.204.107@tcp133 [8/256/0/180] [18727.739419] LNet: Added LNI 192.168.204.107@tcp134 [8/256/0/180] [18729.143888] LNet: Added LNI 192.168.204.107@tcp135 [8/256/0/180] [18738.763110] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [18738.817719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18739.988773] Lustre: 465221:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.107@tcp: can't add NID 192.168.204.107@tcp130: rc = -75 [18740.269780] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [18740.298973] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [18740.389051] Lustre: lustre-MDT0000: new disk, initializing [18740.503275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [18740.533300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [18745.898132] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [18758.974848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18759.222453] Lustre: 465239:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [18759.402124] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [18759.410799] Lustre: Skipped 1 previous similar message [18759.578470] Lustre: lustre-MDT0001: new disk, initializing [18759.626589] Lustre: 466033:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.107@tcp: can't add NID 192.168.204.107@tcp130: rc = -75 [18759.647185] Lustre: 466033:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [18759.818968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [18759.889063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [18759.908327] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [18765.327971] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [18768.469741] Lustre: 466397:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.107@tcp: can't add NID 192.168.204.107@tcp130: rc = -75 [18768.485411] Lustre: 466397:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [18777.304736] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [18785.012665] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [18795.506190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [18796.340408] Lustre: lustre-OST0000: new disk, initializing [18796.356352] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [18796.376475] Lustre: 467207:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [18796.594362] Lustre: 467212:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.107@tcp: can't add NID 192.168.204.107@tcp130: rc = -75 [18796.610325] Lustre: 467212:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [18796.715280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [18801.712886] Lustre: 467421:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.107@tcp: can't add NID 192.168.204.107@tcp130: rc = -75 [18801.721273] Lustre: 467421:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [18801.787030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [18801.802355] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [18802.024787] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [18804.050311] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [18816.936570] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [18827.750386] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18827.753594] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [18827.798077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [18827.817968] Lustre: Skipped 1 previous similar message [18832.512341] Lustre: server umount lustre-OST0000 complete [18838.006681] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18838.014042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18838.030133] Lustre: Skipped 2 previous similar messages [18843.104736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18843.110676] Lustre: Skipped 2 previous similar messages [18843.794498] Lustre: server umount lustre-MDT0000 complete [18848.505787] LustreError: 468569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785795775 with bad export cookie 6369361747834338515 [18848.506508] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [18848.521703] LustreError: 468569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [18849.215425] Lustre: server umount lustre-MDT0001 complete [18861.782911] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [18864.903436] Key type lgssc unregistered [18865.465941] LNet: 469127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [18865.482179] LNetError: 469127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18865.492121] LNet: Removed LNI 192.168.204.107@tcp [18866.527214] LNet: Removed LNI 192.168.204.107@tcp100 [18866.530141] LNetError: 469127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18866.550685] LNetError: 469127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [18866.566206] LNet: Removed LNI 192.168.204.107@tcp101 [18867.615165] LNet: Removed LNI 192.168.204.107@tcp102 [18867.617539] LNetError: 469127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18867.631110] LNetError: 469127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [18867.648445] LNet: Removed LNI 192.168.204.107@tcp103 [18867.654584] LNet: Removed LNI 192.168.204.107@tcp104 [18867.667546] LNet: Removed LNI 192.168.204.107@tcp105 [18867.677174] LNet: Removed LNI 192.168.204.107@tcp106 [18867.678992] LNet: Removed LNI 192.168.204.107@tcp107 [18867.699537] LNet: Removed LNI 192.168.204.107@tcp108 [18867.712554] LNet: Removed LNI 192.168.204.107@tcp109 [18867.715444] LNet: Removed LNI 192.168.204.107@tcp110 [18867.725143] LNet: Removed LNI 192.168.204.107@tcp111 [18867.734157] LNet: Removed LNI 192.168.204.107@tcp112 [18867.739440] LNet: Removed LNI 192.168.204.107@tcp113 [18867.747536] LNet: Removed LNI 192.168.204.107@tcp114 [18867.757649] LNet: Removed LNI 192.168.204.107@tcp115 [18867.760879] LNet: Removed LNI 192.168.204.107@tcp116 [18867.768687] LNet: Removed LNI 192.168.204.107@tcp117 [18867.776662] LNet: Removed LNI 192.168.204.107@tcp118 [18867.782422] LNet: Removed LNI 192.168.204.107@tcp119 [18867.787917] LNet: Removed LNI 192.168.204.107@tcp120 [18867.807703] LNet: Removed LNI 192.168.204.107@tcp121 [18867.816116] LNet: Removed LNI 192.168.204.107@tcp122 [18867.822410] LNet: Removed LNI 192.168.204.107@tcp123 [18867.834098] LNet: Removed LNI 192.168.204.107@tcp124 [18867.839123] LNet: Removed LNI 192.168.204.107@tcp125 [18867.843221] LNet: Removed LNI 192.168.204.107@tcp126 [18867.852785] LNet: Removed LNI 192.168.204.107@tcp127 [18867.858218] LNet: Removed LNI 192.168.204.107@tcp128 [18867.861287] LNet: Removed LNI 192.168.204.107@tcp129 [18867.867520] LNet: Removed LNI 192.168.204.107@tcp130 [18867.870394] LNet: Removed LNI 192.168.204.107@tcp131 [18867.875887] LNet: Removed LNI 192.168.204.107@tcp132 [18867.880394] LNet: Removed LNI 192.168.204.107@tcp133 [18867.886542] LNet: Removed LNI 192.168.204.107@tcp134 [18867.900633] LNet: Removed LNI 192.168.204.107@tcp135 [18869.135985] Key type .llcrypt unregistered [18869.140996] Key type ._llcrypt unregistered [18899.817835] Key type ._llcrypt registered [18899.819608] Key type .llcrypt registered [18899.996477] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [18921.333377] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [18922.970576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [18923.294634] alg: No test for adler32 (adler32-zlib) [18924.609243] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [18925.028682] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [18926.815158] Key type lgssc registered [18928.457773] Lustre: Echo OBD driver; http://www.lustre.org/ [18979.556910] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 18:25:03 (1785795903) [19012.104251] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [19016.934986] Key type lgssc unregistered [19017.501159] LNet: 473240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [19017.520072] LNetError: 473240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19017.537465] LNet: Removed LNI 192.168.204.107@tcp [19019.150057] Key type .llcrypt unregistered [19019.153700] Key type ._llcrypt unregistered [19050.139901] Key type ._llcrypt registered [19050.141167] Key type .llcrypt registered [19050.439811] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [19052.005505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [19052.131929] alg: No test for adler32 (adler32-zlib) [19054.989794] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [19055.704499] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [19057.519647] Key type lgssc registered [19059.350876] Lustre: Echo OBD driver; http://www.lustre.org/ [19071.268996] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [19071.344990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19072.853216] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [19072.905715] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [19073.093100] Lustre: lustre-MDT0000: new disk, initializing [19073.226760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [19073.282744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [19081.029561] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [19095.575788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19095.832142] Lustre: 474455:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [19095.979976] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [19095.991268] Lustre: Skipped 1 previous similar message [19096.142565] Lustre: lustre-MDT0001: new disk, initializing [19096.541794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [19096.607577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [19096.632427] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [19103.431440] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [19120.794507] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [19128.240331] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [19136.459097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [19136.869306] Lustre: lustre-OST0000: new disk, initializing [19136.876945] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [19136.886316] Lustre: 476427:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [19137.003677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [19146.234319] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [19146.815247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [19146.840377] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [19146.973349] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [19159.054855] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [19165.214239] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing lnet_if_list [19173.032726] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [19173.081452] LNet: Added LNI 192.168.204.107@tcp10 [8/256/0/180] [19180.911790] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [19180.933907] LNet: Added LNI 192.168.204.107@tcp11 [8/256/0/180] [19187.533967] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [19187.559578] LNet: Added LNI 192.168.204.107@tcp12 [8/256/0/180] [19193.976928] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [19194.023853] LNet: Added LNI 192.168.204.107@tcp13 [8/256/0/180] [19202.836764] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [19202.935093] LNet: Added LNI 192.168.204.107@tcp14 [8/256/0/180] [19208.762088] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [19208.824556] LNet: Added LNI 192.168.204.107@tcp15 [8/256/0/180] [19214.392278] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [19214.411585] LNet: Added LNI 192.168.204.107@tcp16 [8/256/0/180] [19220.509913] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [19220.568422] LNet: Added LNI 192.168.204.107@tcp17 [8/256/0/180] [19226.538500] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [19226.578189] LNet: Added LNI 192.168.204.107@tcp18 [8/256/0/180] [19234.551407] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [19234.622831] LNet: Added LNI 192.168.204.107@tcp19 [8/256/0/180] [19242.397798] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [19242.459136] LNet: Added LNI 192.168.204.107@tcp20 [8/256/0/180] [19249.811160] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [19249.864033] LNet: Added LNI 192.168.204.107@tcp21 [8/256/0/180] [19259.251549] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [19259.290802] LNet: Added LNI 192.168.204.107@tcp22 [8/256/0/180] [19268.090804] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [19268.146885] LNet: Added LNI 192.168.204.107@tcp23 [8/256/0/180] [19277.419875] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [19277.469931] LNet: Added LNI 192.168.204.107@tcp24 [8/256/0/180] [19286.495623] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [19286.563181] LNet: Added LNI 192.168.204.107@tcp25 [8/256/0/180] [19295.996611] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [19296.090048] LNet: Added LNI 192.168.204.107@tcp26 [8/256/0/180] [19305.524456] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [19305.540630] LNet: Added LNI 192.168.204.107@tcp27 [8/256/0/180] [19311.828544] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [19311.854242] LNet: Added LNI 192.168.204.107@tcp28 [8/256/0/180] [19319.069469] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [19319.088573] LNet: Added LNI 192.168.204.107@tcp29 [8/256/0/180] [19326.037252] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [19326.106277] LNet: Added LNI 192.168.204.107@tcp30 [8/256/0/180] [19336.159964] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [19336.206332] LNet: Added LNI 192.168.204.107@tcp31 [8/256/0/180] [19346.107354] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [19346.167977] LNet: Added LNI 192.168.204.107@tcp32 [8/256/0/180] [19355.528032] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [19355.552665] LNet: Added LNI 192.168.204.107@tcp33 [8/256/0/180] [19364.498221] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [19364.587062] LNet: Added LNI 192.168.204.107@tcp34 [8/256/0/180] [19374.961685] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [19375.012542] LNet: Added LNI 192.168.204.107@tcp35 [8/256/0/180] [19385.032860] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [19385.062832] LNet: Added LNI 192.168.204.107@tcp36 [8/256/0/180] [19396.063917] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [19396.101447] LNet: Added LNI 192.168.204.107@tcp37 [8/256/0/180] [19406.157773] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [19406.210644] LNet: Added LNI 192.168.204.107@tcp38 [8/256/0/180] [19416.009929] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [19416.047141] LNet: Added LNI 192.168.204.107@tcp39 [8/256/0/180] [19425.294893] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [19425.336912] LNet: Added LNI 192.168.204.107@tcp40 [8/256/0/180] [19435.562497] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [19435.640147] LNet: Added LNI 192.168.204.107@tcp41 [8/256/0/180] [19446.339691] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [19446.380702] LNet: Added LNI 192.168.204.107@tcp42 [8/256/0/180] [19458.579114] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [19458.628738] LNet: Added LNI 192.168.204.107@tcp43 [8/256/0/180] [19469.535143] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [19469.609446] LNet: Added LNI 192.168.204.107@tcp44 [8/256/0/180] [19479.316244] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [19479.362833] LNet: Added LNI 192.168.204.107@tcp45 [8/256/0/180] [19491.058042] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [19491.093488] LNet: Added LNI 192.168.204.107@tcp46 [8/256/0/180] [19500.081968] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [19500.099769] LNet: Added LNI 192.168.204.107@tcp47 [8/256/0/180] [19512.282741] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [19512.353284] LNet: Added LNI 192.168.204.107@tcp48 [8/256/0/180] [19525.345865] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [19525.393550] LNet: Added LNI 192.168.204.107@tcp49 [8/256/0/180] [19537.894057] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [19537.918048] LNet: Added LNI 192.168.204.107@tcp50 [8/256/0/180] [19567.075733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19567.147917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19567.163225] Lustre: Skipped 2 previous similar messages [19571.709693] Lustre: server umount lustre-MDT0000 complete [19577.318424] LustreError: 474466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19577.339255] LustreError: 474466:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [19577.411455] LustreError: 474446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785796503 with bad export cookie 13845702895697108127 [19577.426223] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [19577.428062] LustreError: 474446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [19577.933580] Lustre: server umount lustre-MDT0001 complete [19597.727280] Lustre: 473728:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785796508/real 1785796508] req@000000009393e694 x1872542807837952/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785796524 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [19597.777341] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19597.802758] Lustre: Skipped 2 previous similar messages [19602.911573] Lustre: 473725:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785796513/real 1785796513] req@000000007d9d4684 x1872542807838208/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785796529 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [19603.937325] Lustre: 473726:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785796514/real 1785796514] req@000000005c29a58d x1872542807838336/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785796530 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [19608.033136] Lustre: 473728:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785796518/real 1785796518] req@000000000b16ca05 x1872542807838464/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785796534 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [19609.244114] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [19613.160697] Lustre: 473727:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785796523/real 1785796523] req@00000000995f158a x1872542807838592/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785796539 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [19621.525856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19622.432680] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xc025d16d0b9fa77a [19622.465103] Lustre: MGC192.168.204.107@tcp: Connection restored to 0@lo (at 0@lo) [19623.029573] LustreError: 476419:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19623.147160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [19624.171514] LustreError: 500716:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19624.202436] LustreError: 500716:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [19629.540078] LustreError: 476420:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19629.563073] LustreError: 476420:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [19629.846580] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [19631.596079] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [19636.713142] LustreError: 500716:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19636.749243] LustreError: 500716:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [19642.062818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19642.693255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [19647.988928] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [19648.947684] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [19658.429605] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [19664.892720] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [19689.952667] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [19689.965932] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19689.998585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [19694.050797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [19694.052023] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [19694.071152] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19699.184360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [19699.201343] Lustre: Skipped 2 previous similar messages [19704.291097] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [19704.296889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [19704.332621] Lustre: Skipped 1 previous similar message [19704.799767] Lustre: server umount lustre-OST0000 complete [19714.529042] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19714.545274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [19714.548610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19714.573534] Lustre: Skipped 1 previous similar message [19717.819529] Lustre: server umount lustre-MDT0000 complete [19719.649326] LustreError: 500722:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19719.675504] LustreError: 500722:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [19724.364833] LustreError: 476426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785796650 with bad export cookie 13845702895697110906 [19724.374183] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [19724.381152] LustreError: 476426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [19724.961301] Lustre: server umount lustre-MDT0001 complete [19739.562438] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [19743.870404] Key type lgssc unregistered [19744.456545] LNet: 503523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [19744.468803] LNetError: 503523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19744.479357] LNet: Removed LNI 192.168.204.107@tcp [19744.484538] LNet: Removed LNI 192.168.204.107@tcp10 [19744.491331] LNet: Removed LNI 192.168.204.107@tcp11 [19744.500613] LNet: Removed LNI 192.168.204.107@tcp12 [19744.505561] LNet: Removed LNI 192.168.204.107@tcp13 [19744.513221] LNet: Removed LNI 192.168.204.107@tcp14 [19744.518356] LNet: Removed LNI 192.168.204.107@tcp15 [19744.521074] LNet: Removed LNI 192.168.204.107@tcp16 [19744.525562] LNet: Removed LNI 192.168.204.107@tcp17 [19744.531860] LNet: Removed LNI 192.168.204.107@tcp18 [19744.539711] LNet: Removed LNI 192.168.204.107@tcp19 [19744.541552] LNet: Removed LNI 192.168.204.107@tcp20 [19744.549586] LNet: Removed LNI 192.168.204.107@tcp21 [19744.558391] LNet: Removed LNI 192.168.204.107@tcp22 [19744.562678] LNet: Removed LNI 192.168.204.107@tcp23 [19744.567550] LNet: Removed LNI 192.168.204.107@tcp24 [19744.578217] LNet: Removed LNI 192.168.204.107@tcp25 [19744.581523] LNet: Removed LNI 192.168.204.107@tcp26 [19744.588767] LNet: Removed LNI 192.168.204.107@tcp27 [19744.595074] LNet: Removed LNI 192.168.204.107@tcp28 [19744.598172] LNet: Removed LNI 192.168.204.107@tcp29 [19744.611362] LNet: Removed LNI 192.168.204.107@tcp30 [19744.618268] LNet: Removed LNI 192.168.204.107@tcp31 [19744.623985] LNet: Removed LNI 192.168.204.107@tcp32 [19744.627621] LNet: Removed LNI 192.168.204.107@tcp33 [19744.635178] LNet: Removed LNI 192.168.204.107@tcp34 [19744.638883] LNet: Removed LNI 192.168.204.107@tcp35 [19744.643929] LNet: Removed LNI 192.168.204.107@tcp36 [19744.653565] LNet: Removed LNI 192.168.204.107@tcp37 [19744.655633] LNet: Removed LNI 192.168.204.107@tcp38 [19744.664621] LNet: Removed LNI 192.168.204.107@tcp39 [19744.670130] LNet: Removed LNI 192.168.204.107@tcp40 [19744.675844] LNet: Removed LNI 192.168.204.107@tcp41 [19744.682438] LNet: Removed LNI 192.168.204.107@tcp42 [19744.689311] LNet: Removed LNI 192.168.204.107@tcp43 [19744.693377] LNet: Removed LNI 192.168.204.107@tcp44 [19744.701719] LNet: Removed LNI 192.168.204.107@tcp45 [19744.708272] LNet: Removed LNI 192.168.204.107@tcp46 [19744.716336] LNet: Removed LNI 192.168.204.107@tcp47 [19744.723642] LNet: Removed LNI 192.168.204.107@tcp48 [19744.733262] LNet: Removed LNI 192.168.204.107@tcp49 [19744.748292] LNet: Removed LNI 192.168.204.107@tcp50 [19745.977436] Key type .llcrypt unregistered [19745.982838] Key type ._llcrypt unregistered [19783.720128] Key type ._llcrypt registered [19783.729343] Key type .llcrypt registered [19784.035106] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [19803.560697] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [19805.357797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [19805.526554] alg: No test for adler32 (adler32-zlib) [19806.773962] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [19807.151185] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [19808.919215] Key type lgssc registered [19810.890368] Lustre: Echo OBD driver; http://www.lustre.org/ [19859.893050] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [19859.920111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19861.345599] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [19861.393447] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [19861.553300] Lustre: lustre-MDT0000: new disk, initializing [19861.708675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [19861.751640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [19871.070645] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [19892.303392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19892.597107] Lustre: 506727:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [19892.776228] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [19892.783384] Lustre: Skipped 1 previous similar message [19892.887397] Lustre: lustre-MDT0001: new disk, initializing [19893.009711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [19893.052984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [19893.070163] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [19902.155440] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [19916.358469] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [19927.582418] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [19939.407875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [19939.573858] Lustre: *** cfs_fail_loc=90f, val=0*** [19939.616968] Lustre: lustre-OST0000: new disk, initializing [19939.619666] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [19939.623645] Lustre: 508748:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [19939.692036] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [19947.773102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [19947.791475] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [19948.094227] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [19950.001593] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [19968.289724] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [19978.727638] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [19978.739355] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19978.743208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [19978.777719] Lustre: Skipped 1 previous similar message [19983.874259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [19983.887543] Lustre: Skipped 2 previous similar messages [19984.956585] Lustre: server umount lustre-OST0000 complete [19990.502571] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [19990.512516] LustreError: Skipped 1 previous similar message [19990.525151] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19990.559054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19994.083688] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19994.084820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19994.125082] Lustre: Skipped 1 previous similar message [19999.201130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19999.207897] Lustre: Skipped 1 previous similar message [20004.319727] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [20004.689833] Lustre: server umount lustre-MDT0000 complete [20010.254717] LustreError: 506719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785796936 with bad export cookie 13076009463583747702 [20010.280113] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [20010.280417] LustreError: 506719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [20011.203491] Lustre: server umount lustre-MDT0001 complete [20029.313246] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [20033.550105] Key type lgssc unregistered [20034.020544] LNet: 510680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [20034.042565] LNetError: 510680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [20035.111617] LNet: Removed LNI 192.168.204.107@tcp [20036.482461] Key type .llcrypt unregistered [20036.484876] Key type ._llcrypt unregistered [20062.550347] Key type ._llcrypt registered [20062.554036] Key type .llcrypt registered [20062.660745] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [20063.829662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [20064.008471] alg: No test for adler32 (adler32-zlib) [20065.580870] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [20066.032394] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [20067.879225] Key type lgssc registered [20070.025994] Lustre: Echo OBD driver; http://www.lustre.org/ [20080.084995] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [20080.123927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20082.109143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [20088.962074] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [20092.387515] LustreError: 511955:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20092.415207] LustreError: 511955:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [20097.514198] LustreError: 511956:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20100.132376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20100.570717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [20107.272141] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [20118.086288] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [20125.492439] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [20134.840946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [20135.201652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [20143.781450] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [20154.088298] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [20167.075280] loop0: detected capacity change from 0 to 204800000 [20167.399852] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [20167.416656] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [20167.449995] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [20175.368116] loop0: detected capacity change from 0 to 204800000 [20175.611112] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [20175.629145] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [20175.641571] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [20181.612352] loop0: detected capacity change from 0 to 204800000 [20188.690986] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [20188.782770] Lustre: fs15246-OST0000: new disk, initializing [20188.785454] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [20188.789503] Lustre: 515552:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [20188.831646] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [20196.440042] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [20207.045282] loop1: detected capacity change from 0 to 204800000 [20214.369962] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20214.555807] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [20214.641312] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [20214.773844] Lustre: fs15246-MDT0000: new disk, initializing [20215.087683] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [20215.177463] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [20215.378564] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [20216.441326] Lustre: 516679:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000001796f75a x1872543866738048/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1785797158 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [20219.914931] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [20224.567956] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [20224.582133] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [20224.629664] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [20254.690480] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20254.724770] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [20256.735177] Lustre: server umount fs15246-MDT0000 complete [20259.820241] LustreError: 511951:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20264.931523] LustreError: 511951:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20270.079551] LustreError: 511950:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20270.330972] Lustre: server umount fs15246-OST0000 complete [20288.994547] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [20289.002185] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20289.022222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [20290.017027] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [20290.028144] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20290.052761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [20293.183879] Lustre: server umount lustre-OST0000 complete [20300.261178] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [20300.278887] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20300.323786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [20300.329525] Lustre: Skipped 1 previous similar message [20305.891795] LustreError: 511952:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20305.913974] LustreError: 511952:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [20306.030792] Lustre: server umount lustre-MDT0000 complete [20310.562158] LustreError: 511937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785797237 with bad export cookie 3548192827134129578 [20310.563717] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [20310.580398] LustreError: 511937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [20311.148058] Lustre: server umount lustre-MDT0001 complete [20328.070507] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [20333.124792] Key type lgssc unregistered [20333.575365] LNet: 519779:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [20333.578599] LNetError: 519779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [20334.630667] LNet: Removed LNI 192.168.204.107@tcp [20335.938744] Key type .llcrypt unregistered [20335.942168] Key type ._llcrypt unregistered [20362.667414] Key type ._llcrypt registered [20362.670833] Key type .llcrypt registered [20362.883895] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [20363.969068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [20364.387612] alg: No test for adler32 (adler32-zlib) [20367.524641] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [20368.062827] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [20369.861250] Key type lgssc registered [20372.893495] Lustre: Echo OBD driver; http://www.lustre.org/ [20429.836631] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [20448.768446] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [20505.250959] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [20505.294251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20506.876572] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [20506.896072] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [20506.958221] Lustre: lustre-MDT0000: new disk, initializing [20507.010376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [20507.027962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [20514.056345] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [20531.321191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20531.557967] Lustre: 524275:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [20531.696179] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [20531.704885] Lustre: Skipped 1 previous similar message [20531.943763] Lustre: lustre-MDT0001: new disk, initializing [20532.099183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [20532.155176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [20532.172831] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [20539.071324] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [20552.973482] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [20561.157457] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [20571.125967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [20571.523657] Lustre: lustre-OST0000: new disk, initializing [20571.529392] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [20571.536317] Lustre: 526246:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [20571.655742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [20578.955385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [20578.978312] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [20579.145215] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [20579.936227] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [20591.955758] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [20601.234694] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [20601.641910] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [20609.385594] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [20609.528619] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [20615.141146] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [20615.142414] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20615.147339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [20615.152201] LustreError: Skipped 1 previous similar message [20615.183791] Lustre: Skipped 1 previous similar message [20620.262427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [20620.278532] Lustre: Skipped 2 previous similar messages [20625.379241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [20625.406104] Lustre: Skipped 1 previous similar message [20626.402167] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [20626.792851] Lustre: server umount lustre-OST0000 complete [20634.593270] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [20634.606712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20634.641587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [20635.623157] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20637.992361] Lustre: server umount lustre-MDT0000 complete [20643.089168] LustreError: 524267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785797569 with bad export cookie 2251660789430172990 [20643.097362] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [20643.105895] LustreError: 524267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [20643.858689] Lustre: server umount lustre-MDT0001 complete [20682.493926] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [20699.453577] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [20739.404300] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 20340 sec ========= 18:54:23 (1785797663) [20741.610782] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:54:25 (1785797665) === [20746.423170] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:54:30 (1785797670) === [20813.966557] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [20817.839175] Key type lgssc unregistered [20818.351708] LNet: 532943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [20818.365866] LNetError: 532943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [20819.430355] LNet: Removed LNI 192.168.204.107@tcp [20821.433736] Key type .llcrypt unregistered [20821.435055] Key type ._llcrypt unregistered