[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 472120539 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003209] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005019] kvm-guest: setup PV IPIs [ 0.008503] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011162] LSM: Security Framework initializing [ 0.012056] Yama: becoming mindful. [ 0.013044] SELinux: Initializing. [ 0.014077] *** VALIDATE selinux *** [ 0.022511] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027108] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029128] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031112] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033115] *** VALIDATE tmpfs *** [ 0.035066] *** VALIDATE proc *** [ 0.036295] *** VALIDATE cgroup *** [ 0.037011] *** VALIDATE cgroup2 *** [ 0.038278] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040138] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042032] Spectre V2 : User space: Vulnerable [ 0.044006] Speculative Store Bypass: Vulnerable [ 0.047163] debug: unmapping init [mem 0xffffffff8f059000-0xffffffff8f060fff] [ 0.049861] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.050713] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.051022] ... version: 2 [ 0.052011] ... bit width: 48 [ 0.053012] ... generic registers: 4 [ 0.054012] ... value mask: 0000ffffffffffff [ 0.055017] ... max period: 00007fffffffffff [ 0.056013] ... fixed-purpose events: 3 [ 0.057013] ... event mask: 000000070000000f [ 0.058327] rcu: Hierarchical SRCU implementation. [ 0.060413] smp: Bringing up secondary CPUs ... [ 0.061598] x86: Booting SMP configuration: [ 0.062022] .... node #0, CPUs: #1 #2 #3 [ 0.065132] smp: Brought up 1 node, 4 CPUs [ 0.067011] smpboot: Max logical packages: 1 [ 0.068013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.137027] node 0 deferred pages initialised in 67ms [ 0.141087] devtmpfs: initialized [ 0.142211] x86/mm: Memory block size: 128MB [ 0.145049] gcov: version magic: 0x41383552 [ 0.147263] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.150087] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.153353] pinctrl core: initialized pinctrl subsystem [ 0.155208] [ 0.155753] ************************************************************* [ 0.158015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161015] ** ** [ 0.163012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.165017] ** ** [ 0.167010] ** This means that this kernel is built to expose internal ** [ 0.169015] ** IOMMU data structures, which may compromise security on ** [ 0.171014] ** your system. ** [ 0.174014] ** ** [ 0.176012] ** If you see this message and you are not debugging the ** [ 0.178012] ** kernel, report this immediately to your vendor! ** [ 0.181023] ** ** [ 0.183031] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.185011] ************************************************************* [ 0.188502] NET: Registered protocol family 16 [ 0.190434] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.193058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.196071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.199504] cpuidle: using governor menu [ 0.200624] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.202450] PCI: Using configuration type 1 for base access [ 0.205126] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.212124] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.215040] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.217133] cryptd: max_cpu_qlen set to 1000 [ 0.220150] ACPI: Added _OSI(Module Device) [ 0.222020] ACPI: Added _OSI(Processor Device) [ 0.224015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.226028] ACPI: Added _OSI(Processor Aggregator Device) [ 0.233040] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.239474] ACPI: Interpreter enabled [ 0.241053] ACPI: PM: (supports S0 S3 S4 S5) [ 0.242015] ACPI: Using IOAPIC for interrupt routing [ 0.244113] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.247404] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.255939] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.258042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.259011] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.262092] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.266103] acpiphp: Slot [2] registered [ 0.267071] acpiphp: Slot [5] registered [ 0.269103] acpiphp: Slot [6] registered [ 0.270127] acpiphp: Slot [7] registered [ 0.271122] acpiphp: Slot [8] registered [ 0.273156] acpiphp: Slot [9] registered [ 0.275078] acpiphp: Slot [10] registered [ 0.276115] acpiphp: Slot [3] registered [ 0.278088] acpiphp: Slot [4] registered [ 0.279108] acpiphp: Slot [11] registered [ 0.281095] acpiphp: Slot [12] registered [ 0.282090] acpiphp: Slot [13] registered [ 0.284118] acpiphp: Slot [14] registered [ 0.285098] acpiphp: Slot [15] registered [ 0.287129] acpiphp: Slot [16] registered [ 0.288119] acpiphp: Slot [17] registered [ 0.290106] acpiphp: Slot [18] registered [ 0.291082] acpiphp: Slot [19] registered [ 0.293079] acpiphp: Slot [20] registered [ 0.294109] acpiphp: Slot [21] registered [ 0.295075] acpiphp: Slot [22] registered [ 0.296074] acpiphp: Slot [23] registered [ 0.297088] acpiphp: Slot [24] registered [ 0.299078] acpiphp: Slot [25] registered [ 0.300093] acpiphp: Slot [26] registered [ 0.301156] acpiphp: Slot [27] registered [ 0.302069] acpiphp: Slot [28] registered [ 0.303066] acpiphp: Slot [29] registered [ 0.304108] acpiphp: Slot [30] registered [ 0.305075] acpiphp: Slot [31] registered [ 0.306049] PCI host bridge to bus 0000:00 [ 0.307013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.309016] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.311017] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.314040] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.316021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.319034] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.321217] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.324166] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.327405] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.337884] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.342898] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.345016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.347017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.350025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.353402] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.354674] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.356083] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.358662] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.361000] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.372013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.376016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.380671] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.386015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.389016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.404016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.418979] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.430022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.437017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.456019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.469797] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.479015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.483014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.498014] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.505895] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.516015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.523017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.542015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.551028] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.560025] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.570016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.586017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.594267] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.603016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.609016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.622022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.634868] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.637325] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.639312] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.640232] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.641206] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.646099] iommu: Default domain type: Passthrough [ 0.648420] SCSI subsystem initialized [ 0.649086] ACPI: bus type USB registered [ 0.650155] usbcore: registered new interface driver usbfs [ 0.652038] usbcore: registered new interface driver hub [ 0.653057] usbcore: registered new device driver usb [ 0.654101] pps_core: LinuxPPS API ver. 1 registered [ 0.655007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.658058] PTP clock support registered [ 0.660131] EDAC MC: Ver: 3.0.0 [ 0.662137] PCI: Using ACPI for IRQ routing [ 0.663980] NetLabel: Initializing [ 0.666010] NetLabel: domain hash size = 128 [ 0.667010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.668070] NetLabel: unlabeled traffic allowed by default [ 0.670109] vgaarb: loaded [ 0.671365] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.673011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.678403] clocksource: Switched to clocksource kvm-clock [ 0.774197] VFS: Disk quotas dquot_6.6.0 [ 0.776528] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.778740] *** VALIDATE ramfs *** [ 0.780129] *** VALIDATE hugetlbfs *** [ 0.781596] pnp: PnP ACPI init [ 0.783852] pnp: PnP ACPI: found 6 devices [ 0.801993] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.805261] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.806556] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.808039] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.810219] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.812497] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.815387] NET: Registered protocol family 2 [ 0.817862] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.822534] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.825351] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.829422] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.831942] TCP: Hash tables configured (established 65536 bind 65536) [ 0.833994] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.836510] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.838412] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.840959] NET: Registered protocol family 1 [ 0.844253] RPC: Registered named UNIX socket transport module. [ 0.846442] RPC: Registered udp transport module. [ 0.848179] RPC: Registered tcp transport module. [ 0.849792] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.852223] NET: Registered protocol family 44 [ 0.853893] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.856173] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.858292] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.860736] PCI: CLS 0 bytes, default 64 [ 0.862463] Unpacking initramfs... [ 2.206529] debug: unmapping init [mem 0xffff9e40bcc54000-0xffff9e40bffbffff] [ 2.210408] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.212638] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.215629] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.656090] Initialise system trusted keyrings [ 2.657844] Key type blacklist registered [ 2.659311] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.666493] zbud: loaded [ 2.668630] *** VALIDATE nfs *** [ 2.669423] *** VALIDATE nfs4 *** [ 2.670566] pstore: using deflate compression [ 2.673392] Platform Keyring initialized [ 2.760767] NET: Registered protocol family 38 [ 2.762149] Key type asymmetric registered [ 2.763224] Asymmetric key parser 'x509' registered [ 2.764494] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.766610] io scheduler mq-deadline registered [ 2.767606] io scheduler kyber registered [ 2.768677] io scheduler bfq registered [ 2.769874] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.771622] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.773535] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.775184] ACPI: Power Button [PWRF] [ 2.778419] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.784562] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.793760] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.800773] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.832151] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.861023] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.891540] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.896721] Non-volatile memory driver v1.3 [ 2.897967] Linux agpgart interface v0.103 [ 2.928255] virtio_blk virtio1: [vda] 149760 512-byte logical blocks (76.7 MB/73.1 MiB) [ 2.931017] vda: detected capacity change from 0 to 76677120 [ 2.950070] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.953068] vdb: detected capacity change from 0 to 1073741824 [ 2.970559] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.973163] vdc: detected capacity change from 0 to 2621440000 [ 2.987515] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.989347] vdd: detected capacity change from 0 to 2621440000 [ 3.011330] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.013141] vde: detected capacity change from 0 to 4294967296 [ 3.033535] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.035327] vdf: detected capacity change from 0 to 4294967296 [ 3.041690] libphy: Fixed MDIO Bus: probed [ 3.054213] usbcore: registered new interface driver usbserial_generic [ 3.056251] usbserial: USB Serial support registered for generic [ 3.058442] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.062552] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.064034] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.065678] mousedev: PS/2 mouse device common for all mice [ 3.068598] rtc_cmos 00:05: RTC can wake from S4 [ 3.070464] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.071838] rtc_cmos 00:05: registered as rtc0 [ 3.075327] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.077892] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.082858] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.083078] intel_pstate: CPU model not supported [ 3.088424] hid: raw HID events driver (C) Jiri Kosina [ 3.090740] usbcore: registered new interface driver usbhid [ 3.093032] usbhid: USB HID core driver [ 3.094754] drop_monitor: Initializing network drop monitor service [ 3.096845] Initializing XFRM netlink socket [ 3.098443] NET: Registered protocol family 10 [ 3.100718] Segment Routing with IPv6 [ 3.101928] NET: Registered protocol family 17 [ 3.103758] mpls_gso: MPLS GSO support [ 3.109192] RAS: Correctable Errors collector initialized. [ 3.110482] AVX version of gcm_enc/dec engaged. [ 3.111396] AES CTR mode by8 optimization enabled [ 3.171993] sched_clock: Marking stable (3171970701, 0)->(4085660572, -913689871) [ 3.175739] registered taskstats version 1 [ 3.177821] Loading compiled-in X.509 certificates [ 3.180172] zswap: loaded using pool lzo/zbud [ 3.201523] Key type big_key registered [ 3.210554] Key type encrypted registered [ 3.212437] ima: No TPM chip found, activating TPM-bypass! [ 3.214081] ima: Allocated hash algorithm: sha1 [ 3.215894] ima: No architecture policies found [ 3.217546] evm: Initialising EVM extended attributes: [ 3.219367] evm: security.selinux [ 3.220131] evm: security.ima [ 3.220829] evm: security.capability [ 3.221574] evm: HMAC attrs: 0x1 [ 3.222961] rtc_cmos 00:05: setting system clock to 2026-09-03 12:06:26 UTC (1788437186) [ 3.226676] debug: unmapping init [mem 0xffffffff90003000-0xffffffff901fffff] [ 3.228678] debug: unmapping init [mem 0xffffffff8ed82000-0xffffffff8f058fff] [ 3.235067] Write protecting the kernel read-only data: 28672k [ 3.237686] debug: unmapping init [mem 0xffffffff8d403000-0xffffffff8d5fffff] [ 3.239408] debug: unmapping init [mem 0xffffffff8dd14000-0xffffffff8ddfffff] [ 3.264869] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.273538] systemd[1]: Detected virtualization kvm. [ 3.275519] systemd[1]: Detected architecture x86-64. [ 3.277448] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.305377] systemd[1]: No hostname configured. [ 3.307380] systemd[1]: Set hostname to . [ 3.309790] random: systemd: uninitialized urandom read (16 bytes read) [ 3.312634] systemd[1]: Initializing machine ID from random generator. [ 3.424495] random: systemd: uninitialized urandom read (16 bytes read) [ 3.427545] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.433452] random: systemd: uninitialized urandom read (16 bytes read) [ 3.436635] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.440588] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.967148] device-mapper: uevent: version 1.0.3 [ 3.969495] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.592054] virtio_net virtio0 ens2: renamed from eth0 [ 4.606398] random: fast init done [ 4.624649] scsi host0: ata_piix [ 4.626832] scsi host1: ata_piix [ 4.628920] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.631346] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.353257] dracut-initqueue[576]: RTNETLINK answers: File exists [ 9.725748] random: crng init done [ 9.726858] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 9.994427] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.083194] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.278547] SELinux: Disabled at runtime. [ 11.333957] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.340772] systemd[1]: Detected virtualization kvm. [ 11.342328] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.782432] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.785751] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.793982] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.797840] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.801268] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.808562] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.814198] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Paths. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... [ 11.939583] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.237908] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.542952] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.559877] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.654586] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.670246] EDAC sbridge: Ver: 1.1.2 [ 13.992164] Key type dns_resolver registered [ 14.279240] NFS: Registering the id_resolver key type [ 14.280355] Key type id_resolver registered [ 14.281217] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg146-server login: [ 62.096687] libcfs: loading out-of-tree module taints kernel. [ 62.131891] Key type ._llcrypt registered [ 62.133996] Key type .llcrypt registered [ 62.281915] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 80.772572] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 82.473861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 82.492157] alg: No test for adler32 (adler32-zlib) [ 84.178880] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 85.037594] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 86.856908] Key type lgssc registered [ 88.405311] Lustre: Echo OBD driver; http://www.lustre.org/ [ 91.590437] hrtimer: interrupt took 27455244 ns [ 108.700298] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 149.550769] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 163.650798] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 163.710962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 165.280240] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 165.334768] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 165.473621] Lustre: lustre-MDT0000: new disk, initializing [ 165.619419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 165.647106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 170.189457] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 184.900346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 185.044892] Lustre: 6507:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 185.087548] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 185.091666] Lustre: Skipped 1 previous similar message [ 185.166751] Lustre: lustre-MDT0001: new disk, initializing [ 185.245521] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 185.285857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 185.298232] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 190.123965] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 195.508687] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 206.237253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 206.554545] Lustre: lustre-OST0000: new disk, initializing [ 206.561113] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 206.573973] Lustre: 8445:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 206.651765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 214.129218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 214.158855] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 214.268703] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 214.275360] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 229.360224] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 229.497721] Lustre: lustre-OST0001: new disk, initializing [ 229.500860] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 229.505445] Lustre: 9518:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 229.556793] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 236.110413] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 236.116143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 236.128936] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 236.188358] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 248.890642] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 254.728334] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 261.214696] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing check_logdir /tmp/testlogs/ [ 266.653800] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing yml_node [ 272.070857] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 274.650058] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 276.997304] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 278.830358] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 08:11:01 EDT 2026 [ 294.966584] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 296.347737] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 313.319024] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 313.320610] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 313.328435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 313.356352] Lustre: Skipped 2 previous similar messages [ 318.432897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 318.437088] Lustre: Skipped 3 previous similar messages [ 323.557487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 323.563940] Lustre: Skipped 3 previous similar messages [ 327.136438] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 327.439500] Lustre: server umount lustre-MDT0000 complete [ 331.542919] LustreError: 7463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437514 with bad export cookie 3336066987375946150 [ 331.549933] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 331.556849] LustreError: 7463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 333.794608] LustreError: 6513:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 333.796123] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 333.798285] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 333.812766] LustreError: 6513:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 333.882161] Lustre: Skipped 2 previous similar messages [ 336.801653] LustreError: 11767:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 336.829843] LustreError: 11767:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 338.010745] Lustre: server umount lustre-MDT0001 complete [ 342.338606] Lustre: server umount lustre-OST0000 complete [ 346.366369] Lustre: server umount lustre-OST0001 complete [ 353.503470] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 363.313909] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 396.090389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 396.324204] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 396.350837] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 396.407379] Lustre: lustre-MDT0000: new disk, initializing [ 396.467523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 396.484562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 400.734943] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 411.003911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 411.148945] Lustre: 15074:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 411.227383] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 411.231724] Lustre: Skipped 1 previous similar message [ 411.387502] Lustre: lustre-MDT0001: new disk, initializing [ 411.587116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 411.695641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 411.725189] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 418.064238] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 430.262853] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 436.947226] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 444.414318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 444.785480] Lustre: lustre-OST0000: new disk, initializing [ 444.789573] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 444.804759] Lustre: 17040:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 444.928889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 446.139679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 446.166644] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 446.299711] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 451.434170] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 461.734617] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 466.773191] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 467.046440] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 472.004373] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 472.253267] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 477.163639] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 477.165197] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 477.170829] LustreError: Skipped 1 previous similar message [ 477.175390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 477.175399] Lustre: Skipped 3 previous similar messages [ 477.215116] Lustre: Skipped 1 previous similar message [ 480.660452] Lustre: server umount lustre-OST0000 complete [ 487.393551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 487.416423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 487.429269] Lustre: Skipped 2 previous similar messages [ 490.753607] Lustre: server umount lustre-MDT0000 complete [ 492.514279] LustreError: 15081:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 492.542078] LustreError: 15081:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 494.203375] LustreError: 17039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437677 with bad export cookie 3336066987375947683 [ 494.211984] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 494.567592] Lustre: server umount lustre-MDT0001 complete [ 498.873975] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 08:14:41 (1788437681) [ 501.390909] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 502.900924] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 08:14:45 (1788437685) [ 509.737373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 510.269106] LustreError: 19449:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 510.377680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 515.019802] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 515.557115] LustreError: 19450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 524.085350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 529.616423] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 537.141826] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 542.220692] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 549.050502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 555.745293] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 563.917120] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 574.967764] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 574.970981] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 574.988659] Lustre: Skipped 1 previous similar message [ 574.994269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 577.279128] Lustre: server umount lustre-OST0000 complete [ 585.697746] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 585.703517] LustreError: Skipped 1 previous similar message [ 585.706558] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 585.715778] Lustre: Skipped 1 previous similar message [ 587.810737] Lustre: server umount lustre-MDT0000 complete [ 591.333898] LustreError: 19446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 591.356752] LustreError: 19446:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 592.115537] LustreError: 19431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437775 with bad export cookie 3336066987375948565 [ 592.121058] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 592.126532] LustreError: 19431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 592.433789] Lustre: server umount lustre-MDT0001 complete [ 605.091776] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 608.709195] Key type lgssc unregistered [ 609.032094] LNet: 22980:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 609.040099] LNetError: 22980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 609.051462] LNet: Removed LNI 192.168.201.146@tcp [ 609.984200] Key type .llcrypt unregistered [ 609.986331] Key type ._llcrypt unregistered [ 634.621929] Key type ._llcrypt registered [ 634.624704] Key type .llcrypt registered [ 634.728664] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 635.962964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 636.091525] alg: No test for adler32 (adler32-zlib) [ 637.227540] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 637.609772] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 639.376210] Key type lgssc registered [ 641.071891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 650.715815] loop: module loaded [ 650.822229] loop0: detected capacity change from 0 to 204800000 [ 651.070026] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 651.079160] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 651.323968] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 658.244162] loop0: detected capacity change from 0 to 204800000 [ 658.442751] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 658.454950] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 658.474717] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 664.307553] loop0: detected capacity change from 0 to 204800000 [ 669.665930] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 669.691514] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 670.940153] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 671.002967] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 671.085465] Lustre: lustre-MDT0000: new disk, initializing [ 671.183887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 671.211484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 675.968564] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 684.863178] loop1: detected capacity change from 0 to 204800000 [ 690.620920] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 690.910281] Lustre: lustre-OST07c6: new disk, initializing [ 690.918275] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 690.927594] Lustre: Skipped 1 previous similar message [ 690.941615] Lustre: 26165:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 691.056426] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 696.938176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 696.945528] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 696.984753] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 698.142576] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 703.286409] Lustre: 25154:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 703.296971] Lustre: 25154:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 703.304480] Lustre: 25154:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 703.312471] Lustre: 25154:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 703.321543] Lustre: 25154:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 703.330469] Lustre: 25154:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 717.296850] Lustre: 25152:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 717.311481] Lustre: 25152:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 717.318737] Lustre: 25152:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 717.324607] Lustre: 25152:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 717.333256] Lustre: 25152:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 717.338622] Lustre: 25152:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 722.021908] Lustre: 25152:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 722.034131] Lustre: 25152:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 722.039597] Lustre: 25152:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 722.047479] Lustre: 25152:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.055984] Lustre: 25152:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 722.071202] Lustre: 25152:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.083291] Lustre: 25152:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 722.094495] Lustre: 25152:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.104478] Lustre: 25152:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 722.112940] Lustre: 25152:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.120046] Lustre: 25152:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 722.124704] Lustre: 25152:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.350840] Lustre: 25152:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 725.371297] Lustre: 25152:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 725.380510] Lustre: 25152:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 725.389255] Lustre: 25152:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.400579] Lustre: 25152:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 725.403739] Lustre: 25152:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.409827] Lustre: 25152:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 725.418693] Lustre: 25152:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.422975] Lustre: 25152:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 725.425543] Lustre: 25152:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.428742] Lustre: 25152:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 725.431705] Lustre: 25152:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 730.007811] Lustre: 25152:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 730.019366] Lustre: 25152:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 730.026352] Lustre: 25152:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 730.035961] Lustre: 25152:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 730.045187] Lustre: 25152:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 730.051913] Lustre: 25152:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 730.061934] Lustre: 25152:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 730.068194] Lustre: 25152:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 730.072291] Lustre: 25152:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 730.076957] Lustre: 25152:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 730.082336] Lustre: 25152:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 730.088962] Lustre: 25152:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 739.718539] Lustre: 25152:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 739.727183] Lustre: 25152:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 739.739272] Lustre: 25152:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 739.749179] Lustre: 25152:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 739.772396] Lustre: 25152:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 739.781359] Lustre: 25152:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 739.797341] Lustre: 25152:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 739.804954] Lustre: 25152:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 739.813043] Lustre: 25152:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 739.818583] Lustre: 25152:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 739.825986] Lustre: 25152:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 739.843667] Lustre: 25152:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 787.938576] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 787.959037] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 793.059411] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 793.263852] Lustre: server umount lustre-OST07c6 complete [ 802.180969] Lustre: server umount lustre-MDT0000 complete [ 816.500904] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 820.114187] Key type lgssc unregistered [ 820.517942] LNet: 28718:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 820.524649] LNetError: 28718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 820.544884] LNet: Removed LNI 192.168.201.146@tcp [ 821.717268] Key type .llcrypt unregistered [ 821.719280] Key type ._llcrypt unregistered [ 843.863798] Key type ._llcrypt registered [ 843.867436] Key type .llcrypt registered [ 843.948313] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 845.118698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 845.253732] alg: No test for adler32 (adler32-zlib) [ 846.509592] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 846.815600] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 848.728635] Key type lgssc registered [ 850.141368] Lustre: Echo OBD driver; http://www.lustre.org/ [ 857.738914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 857.750727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 859.214750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 863.769366] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 869.349674] LustreError: 29987:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 869.360994] LustreError: 29987:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 872.659056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 873.243528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 877.932491] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 885.903512] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 891.211849] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 898.192898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 898.585766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 904.753124] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 905.017157] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 911.938324] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 930.273734] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 930.289459] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 930.297767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 933.346156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 933.353527] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 933.360451] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 934.577285] Lustre: server umount lustre-OST0000 complete [ 938.466427] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 938.478447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 938.488282] Lustre: Skipped 2 previous similar messages [ 943.586277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 943.592678] Lustre: Skipped 1 previous similar message [ 948.706131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 948.718234] Lustre: Skipped 1 previous similar message [ 952.288237] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 952.483658] Lustre: server umount lustre-MDT0000 complete [ 952.811082] LustreError: 29986:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 952.828767] LustreError: 29986:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 955.493660] LustreError: 29967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438138 with bad export cookie 2152187499982680321 [ 955.504067] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 955.513294] LustreError: 29967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 955.704065] Lustre: server umount lustre-MDT0001 complete [ 966.151734] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 968.926536] Key type lgssc unregistered [ 969.286201] LNet: 33726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 969.292321] LNetError: 33726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 969.323087] LNet: Removed LNI 192.168.201.146@tcp [ 970.122126] Key type .llcrypt unregistered [ 970.125103] Key type ._llcrypt unregistered [ 988.131874] Key type ._llcrypt registered [ 988.133538] Key type .llcrypt registered [ 988.273391] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 989.732469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 989.757748] alg: No test for adler32 (adler32-zlib) [ 990.801447] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 991.046065] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 992.712263] Key type lgssc registered [ 993.866981] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1001.437443] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1001.455964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1002.952770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1006.247689] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1013.220838] LustreError: 34982:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1013.247761] LustreError: 34982:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1013.853590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1014.300728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1019.625391] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1028.101603] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1034.184240] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1041.272345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1041.601153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1048.033484] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1048.753528] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1055.130734] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1069.551409] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1069.564433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1069.576795] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1073.727920] Lustre: server umount lustre-OST0000 complete [ 1079.789125] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1079.795253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1079.806047] Lustre: Skipped 2 previous similar messages [ 1079.824932] Lustre: Skipped 2 previous similar messages [ 1083.938392] Lustre: server umount lustre-MDT0000 complete [ 1084.901684] LustreError: 34978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1084.915632] LustreError: 34978:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1086.918549] LustreError: 34963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438270 with bad export cookie 36529858310538459 [ 1086.929082] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1086.933055] LustreError: 34963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1087.222186] Lustre: server umount lustre-MDT0001 complete [ 1097.245394] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 1099.181046] Key type lgssc unregistered [ 1099.383690] LNet: 38537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1099.389150] LNetError: 38537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1099.406185] LNet: Removed LNI 192.168.201.146@tcp [ 1100.355212] Key type .llcrypt unregistered [ 1100.361220] Key type ._llcrypt unregistered [ 1127.012213] Key type ._llcrypt registered [ 1127.014785] Key type .llcrypt registered [ 1127.103768] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 1128.612046] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1128.674812] alg: No test for adler32 (adler32-zlib) [ 1129.742480] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1130.286748] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 1132.113885] Key type lgssc registered [ 1133.932254] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1142.477368] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1142.495469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1143.926307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1149.423733] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1154.543491] LustreError: 39950:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1154.571954] LustreError: 39950:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1157.952580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1158.304646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1162.762563] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1171.441247] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1176.851571] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1183.699324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1184.207053] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1189.357916] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1190.594217] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1197.965358] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1208.809044] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1208.814715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1208.828159] Lustre: Skipped 1 previous similar message [ 1208.842574] Lustre: Skipped 1 previous similar message [ 1210.043290] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.46@tcp (stopping) [ 1213.375136] Lustre: server umount lustre-MDT0000 complete [ 1213.925293] LustreError: 39949:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1213.941538] LustreError: 39949:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1215.131598] LustreError: 39945:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1217.563109] LustreError: 39930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438400 with bad export cookie 5549422368517927001 [ 1217.573477] LustreError: 39930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1217.575890] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1219.041416] LustreError: 39946:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1219.042105] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1219.077940] Lustre: Skipped 1 previous similar message [ 1219.095070] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1222.051045] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1222.059626] Lustre: Skipped 1 previous similar message [ 1224.167628] Lustre: server umount lustre-MDT0001 complete [ 1240.694047] Lustre: server umount lustre-OST0000 complete [ 1259.542181] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 1262.543223] Key type lgssc unregistered [ 1262.845612] LNet: 43807:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1262.859700] LNetError: 43807:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1262.886535] LNet: Removed LNI 192.168.201.146@tcp [ 1263.657961] Key type .llcrypt unregistered [ 1263.660214] Key type ._llcrypt unregistered [ 1285.606246] Key type ._llcrypt registered [ 1285.608918] Key type .llcrypt registered [ 1285.721824] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 1286.776204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1287.059060] alg: No test for adler32 (adler32-zlib) [ 1288.152626] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1288.389224] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 1290.080164] Key type lgssc registered [ 1291.107113] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1299.898486] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1299.944446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1301.579212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1306.681854] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1311.716532] LustreError: 45224:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1311.731589] LustreError: 45224:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1315.413431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1315.728300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1319.964996] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1327.575449] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1333.424899] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1340.617491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1340.875709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1346.064183] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1347.817690] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1356.040521] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1366.498795] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1366.507200] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1366.542161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1369.269248] Lustre: lustre-OST0000: Not available for connect from 192.168.201.46@tcp (stopping) [ 1370.598467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1370.607547] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1370.624906] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1372.715416] Lustre: server umount lustre-OST0000 complete [ 1391.762085] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1391.788330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1391.798572] Lustre: Skipped 1 previous similar message [ 1395.809828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1395.818452] Lustre: Skipped 2 previous similar messages [ 1398.272142] Lustre: server umount lustre-MDT0000 complete [ 1401.313398] LustreError: 45224:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1401.327452] LustreError: 45224:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1402.031131] LustreError: 45205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438585 with bad export cookie 15830323014800412450 [ 1402.034837] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1402.042980] LustreError: 45205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1408.496308] Lustre: server umount lustre-MDT0001 complete [ 1423.124436] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 1426.669604] Key type lgssc unregistered [ 1427.074049] LNet: 48935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1427.082531] LNetError: 48935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1427.119132] LNet: Removed LNI 192.168.201.146@tcp [ 1428.197694] Key type .llcrypt unregistered [ 1428.204135] Key type ._llcrypt unregistered [ 1455.236974] Key type ._llcrypt registered [ 1455.239886] Key type .llcrypt registered [ 1455.405648] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 1456.769082] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1456.885130] alg: No test for adler32 (adler32-zlib) [ 1458.042281] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1458.381674] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 1460.082358] Key type lgssc registered [ 1461.394188] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1470.312601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1470.325717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1471.824436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1476.617867] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1482.219198] LustreError: 50347:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1482.232553] LustreError: 50347:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1484.618395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1485.021512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1490.036184] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1497.481433] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1502.542265] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1509.499188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1509.883394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1517.084539] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1519.046194] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1527.885395] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1534.367722] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1538.087641] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1550.336293] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-03 8h32m12s [ 1552.236742] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1556.207980] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.46@tcp (stopping) [ 1556.961367] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1556.976269] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1557.007963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1558.116957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1558.128306] Lustre: Skipped 5 previous similar messages [ 1560.987103] Lustre: server umount lustre-MDT0000 complete [ 1561.305887] LustreError: 52797:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1564.665029] LustreError: 50344:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1564.688880] LustreError: 50344:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1565.123223] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1567.388677] LustreError: 50343:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1567.425558] LustreError: 50343:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1571.853344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1571.981315] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1572.319961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1572.416694] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1573.368334] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1577.795922] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1581.961829] Lustre: DEBUG MARKER: Wait for df (36668) ... [ 1584.117903] Lustre: DEBUG MARKER: done [ 1591.777367] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1591.783327] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1591.798311] Lustre: Skipped 2 previous similar messages [ 1591.804595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1592.823993] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1592.848218] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1595.754263] Lustre: server umount lustre-OST0000 complete [ 1604.070132] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1604.082055] Lustre: Skipped 1 previous similar message [ 1604.086354] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1604.086861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1604.102034] Lustre: Skipped 4 previous similar messages [ 1614.304134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1614.566396] Lustre: server umount lustre-MDT0000 complete [ 1617.381866] LustreError: 50344:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1617.409636] LustreError: 50344:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1617.850313] LustreError: 52190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438801 with bad export cookie 15993351041779407439 [ 1617.852544] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1617.857059] LustreError: 52190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1618.255240] Lustre: server umount lustre-MDT0001 complete [ 1630.278766] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 1633.576882] Key type lgssc unregistered [ 1634.061831] LNet: 55275:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1634.075694] LNetError: 55275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1634.115310] LNet: Removed LNI 192.168.201.146@tcp [ 1635.445072] Key type .llcrypt unregistered [ 1635.447655] Key type ._llcrypt unregistered [ 1678.996973] Key type ._llcrypt registered [ 1679.000584] Key type .llcrypt registered [ 1679.172842] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 1680.824344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1680.854425] alg: No test for adler32 (adler32-zlib) [ 1681.983594] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1682.298909] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 1683.984179] Key type lgssc registered [ 1685.291944] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1693.443803] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1693.462223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1694.823113] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1694.858547] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1695.053342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1700.371975] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1707.842927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1707.939120] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1707.979488] Lustre: 57481:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1708.460818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1714.385118] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1723.035469] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1729.186616] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1736.643304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1736.861260] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1737.194464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1743.595225] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1747.195287] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1752.210465] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1758.360237] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-03 8h35m41s [ 1759.885398] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1767.791957] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1771.538341] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1771.603720] LustreError: 57490:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1791.679255] Lustre: lustre-MDT0000: Client 7989afab-74de-4007-aacb-46e74a871fa0 (at 192.168.201.46@tcp) reconnecting [ 1811.656103] LustreError: 57490:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1813.817992] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1816.632431] Lustre: DEBUG MARKER: done [ 1824.226587] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1824.239033] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1824.241547] LustreError: Skipped 1 previous similar message [ 1824.242810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1824.272012] Lustre: Skipped 1 previous similar message [ 1824.610268] Lustre: server umount lustre-OST0000 complete [ 1829.353748] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1829.366059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1829.376058] Lustre: Skipped 2 previous similar messages [ 1830.437436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1830.446308] Lustre: Skipped 1 previous similar message [ 1834.579271] Lustre: server umount lustre-MDT0000 complete [ 1837.768044] LustreError: 57473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439021 with bad export cookie 3100439483558426988 [ 1837.773792] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1837.777307] LustreError: 57473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1838.015113] Lustre: server umount lustre-MDT0001 complete [ 1848.952428] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 1851.247105] Key type lgssc unregistered [ 1851.460658] LNet: 61819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1851.465423] LNetError: 61819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1851.478281] LNet: Removed LNI 192.168.201.146@tcp [ 1852.017659] Key type .llcrypt unregistered [ 1852.019382] Key type ._llcrypt unregistered [ 1894.509572] Key type ._llcrypt registered [ 1894.512405] Key type .llcrypt registered [ 1894.671592] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 1895.811687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1895.919124] alg: No test for adler32 (adler32-zlib) [ 1897.320636] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1897.750153] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 1899.536205] Key type lgssc registered [ 1900.838810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1904.694145] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1904.711134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1905.916414] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1905.934553] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1906.078090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1907.091239] Lustre: Failing over lustre-MDT0000 [ 1907.410350] Lustre: server umount lustre-MDT0000 complete [ 1913.215311] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 08:38:16 (1788439096) [ 1919.032391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1919.429307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1924.079986] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1932.120245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1932.226685] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1932.251776] Lustre: 64591:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1932.500113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1936.758561] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1943.539546] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1948.965289] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1955.230306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1955.403900] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1955.572359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1962.104929] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1962.637224] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1968.908164] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1975.530148] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1983.460922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1983.475429] Lustre: Skipped 1 previous similar message [ 1983.492152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1983.495832] Lustre: Skipped 1 previous similar message [ 1988.583918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1988.593241] Lustre: Skipped 2 previous similar messages [ 1993.699425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1993.711075] Lustre: Skipped 2 previous similar messages [ 1996.256172] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1996.650559] Lustre: server umount lustre-MDT0000 complete [ 2000.835157] LustreError: 64583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439184 with bad export cookie 16177670563375107806 [ 2000.837078] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2000.844098] LustreError: 64583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2001.218829] Lustre: server umount lustre-MDT0001 complete [ 2004.842698] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2009.500808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2019.373144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2020.321669] Lustre: 63517:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788439187/real 1788439187] req@00000000031f2fd0 x1875314296791936/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788439203 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2020.352161] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2020.366471] Lustre: Skipped 1 previous similar message [ 2022.432346] Lustre: 63517:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788439189/real 1788439189] req@000000008791b12c x1875314296792064/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788439205 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2025.440156] Lustre: 63514:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788439192/real 1788439192] req@00000000491db497 x1875314296792192/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788439208 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2026.750648] LustreError: 66440:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2026.774112] LustreError: 66440:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2026.836043] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2026.926142] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2027.808093] Lustre: 63514:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788439194/real 1788439194] req@00000000a93cfafb x1875314296792448/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788439210 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2028.900769] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2028.901453] LustreError: 66441:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2029.932896] LustreError: 66440:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2029.952535] LustreError: 66440:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2031.214704] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2031.968112] Lustre: 63516:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788439199/real 1788439199] req@00000000d967299d x1875314296792576/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788439215 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2035.169606] LustreError: 68358:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2035.193738] LustreError: 68358:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2039.017189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2039.412641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2043.336813] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2044.931825] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2050.443779] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2056.848360] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2064.868541] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2064.878776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2064.904182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2069.757634] Lustre: server umount lustre-MDT0000 complete [ 2070.506960] LustreError: 66440:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2070.527403] LustreError: 66440:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2073.523945] LustreError: 65327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439256 with bad export cookie 16177670563375110081 [ 2073.533101] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2073.538845] LustreError: 65327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2073.871193] Lustre: server umount lustre-MDT0001 complete [ 2080.644661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2087.640161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2090.976183] Lustre: 63514:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788439258/real 1788439258] req@000000006ef2a1fb x1875314296818688/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788439274 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2090.996558] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2091.005237] Lustre: Skipped 2 previous similar messages [ 2098.494933] LustreError: 66441:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2098.588690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2098.626888] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2104.342760] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2113.976666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2119.350664] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2119.667120] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2128.396966] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2134.589197] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2145.252772] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2145.266466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2145.287634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2145.293513] Lustre: Skipped 3 previous similar messages [ 2147.788827] Lustre: server umount lustre-MDT0000 complete [ 2151.401154] LustreError: 71043:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2151.414103] LustreError: 71043:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 2151.495605] LustreError: 64584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439334 with bad export cookie 16177670563375111460 [ 2151.508895] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2151.508954] LustreError: 64584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2151.756994] Lustre: server umount lustre-MDT0001 complete [ 2158.786700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2162.172326] Lustre: DEBUG MARKER: files compared the same [ 2174.969159] Lustre: server umount lustre-OST0000 complete [ 2191.596236] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 2194.470172] Key type lgssc unregistered [ 2194.736652] LNet: 74319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2194.741212] LNetError: 74319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2194.767324] LNet: Removed LNI 192.168.201.146@tcp [ 2195.698053] Key type .llcrypt unregistered [ 2195.701786] Key type ._llcrypt unregistered [ 2219.735928] Key type ._llcrypt registered [ 2219.737426] Key type .llcrypt registered [ 2219.907875] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 2221.252639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2221.308385] alg: No test for adler32 (adler32-zlib) [ 2222.323440] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2222.522856] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 2224.176327] Key type lgssc registered [ 2225.304502] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2232.514991] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2232.560610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2234.038276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2238.464929] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug +malloc all [ 2244.588438] LustreError: 75731:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2244.613853] LustreError: 75731:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2246.574703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2246.951181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2251.981842] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug +malloc all [ 2259.079037] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2264.096799] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2270.779046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2271.031459] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2277.252154] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug +malloc all [ 2279.222896] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2284.268385] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2294.753718] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2294.755158] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2294.762067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2294.775486] Lustre: Skipped 1 previous similar message [ 2294.793072] Lustre: Skipped 1 previous similar message [ 2299.474853] Lustre: server umount lustre-OST0000 complete [ 2303.460441] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2303.472512] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2303.484076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2304.996956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2305.006362] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2309.717882] Lustre: server umount lustre-MDT0000 complete [ 2310.114589] LustreError: 76459:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2310.133679] LustreError: 76459:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2313.276371] LustreError: 75711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439496 with bad export cookie 12678115186869146414 [ 2313.283956] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2313.561145] Lustre: server umount lustre-MDT0001 complete [ 2324.349370] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 2326.906490] Key type lgssc unregistered [ 2327.156715] LNet: 79267:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2327.162934] LNetError: 79267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2327.175799] LNet: Removed LNI 192.168.201.146@tcp [ 2328.018313] Key type .llcrypt unregistered [ 2328.020900] Key type ._llcrypt unregistered [ 2351.139766] Key type ._llcrypt registered [ 2351.142898] Key type .llcrypt registered [ 2351.262969] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 2352.401878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2352.449264] alg: No test for adler32 (adler32-zlib) [ 2353.682371] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2354.041527] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 2355.760195] Key type lgssc registered [ 2356.760578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2364.901671] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2364.920512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2380.513171] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2385.633226] LustreError: 80654:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.146@tcp: failed processing log, type 4: rc = -110 [ 2412.258942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2419.172566] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2428.397934] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2438.939443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2439.001237] Lustre: *** cfs_fail_loc=706, val=0*** [ 2439.445784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2439.497465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2443.613235] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2444.772751] LustreError: 80680:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2444.788154] LustreError: 80680:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2446.828396] LustreError: 81740:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2446.855307] LustreError: 81740:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2451.944281] LustreError: 80680:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2451.967921] LustreError: 80680:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2452.787742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2453.288713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2458.427167] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2465.711841] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2471.861712] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2475.490051] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2475.500975] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2475.516869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2479.088744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2479.109035] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2479.120705] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2480.305291] Lustre: server umount lustre-OST0000 complete [ 2484.193706] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2484.199431] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2484.200748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2484.200754] Lustre: Skipped 1 previous similar message [ 2484.202990] Lustre: Skipped 1 previous similar message [ 2489.314423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2489.322782] Lustre: Skipped 2 previous similar messages [ 2490.023117] Lustre: server umount lustre-MDT0000 complete [ 2494.350660] LustreError: 80660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439677 with bad export cookie 10580066691074462775 [ 2494.362859] LustreError: 80660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2494.366528] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2494.691680] Lustre: server umount lustre-MDT0001 complete [ 2506.702645] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 2509.963727] Key type lgssc unregistered [ 2510.244230] LNet: 84270:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2510.251802] LNetError: 84270:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2511.276530] LNet: Removed LNI 192.168.201.146@tcp [ 2512.345688] Key type .llcrypt unregistered [ 2512.347576] Key type ._llcrypt unregistered [ 2538.572259] Key type ._llcrypt registered [ 2538.575426] Key type .llcrypt registered [ 2538.673237] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 2539.796316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2539.857658] alg: No test for adler32 (adler32-zlib) [ 2540.981491] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2541.249658] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 2542.936424] Key type lgssc registered [ 2544.327785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2553.111693] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2553.132514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2559.410944] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2568.003251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2568.540739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2573.685880] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2573.797598] LustreError: 86373:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2573.814235] LustreError: 86373:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2577.893858] LustreError: 86376:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2577.917296] LustreError: 86376:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2583.028554] LustreError: 86377:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2583.050072] LustreError: 86377:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2583.353343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2583.722055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2589.158202] LustreError: 86371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2589.181532] LustreError: 86371:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2590.816114] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2593.250818] LustreError: 86376:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2593.274379] LustreError: 86376:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2599.358533] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2599.371678] Lustre: MGS: Client d56f0cb9-2dda-4c37-9fa8-9aa40dd48d79 (at 0@lo) reconnecting [ 2599.398940] LustreError: 86387:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2599.590574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2599.689072] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2604.843854] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2620.386304] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2620.405346] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2620.414044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2620.417196] Lustre: Skipped 1 previous similar message [ 2623.564520] Lustre: server umount lustre-OST0000 complete [ 2630.628686] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2630.629658] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2630.645273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2630.657751] Lustre: Skipped 1 previous similar message [ 2630.669230] Lustre: Skipped 2 previous similar messages [ 2633.154352] Lustre: server umount lustre-MDT0000 complete [ 2635.749387] LustreError: 87394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2635.786812] LustreError: 87394:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2638.248356] Lustre: server umount lustre-MDT0001 complete [ 2642.611916] Lustre: server umount MGS complete [ 2655.597605] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 08:50:38 (1788439838) [ 2689.850774] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 2699.686854] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 2745.057833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2750.315361] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2760.250197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2760.510591] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2760.571946] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2760.637136] Lustre: lustre-MDT0001: new disk, initializing [ 2760.752965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2760.789358] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2760.798759] Lustre: 93546:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2760.810337] LustreError: 93546:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2764.716745] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2775.312389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2775.669509] Lustre: lustre-OST0000: new disk, initializing [ 2775.673683] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2775.680648] Lustre: 94313:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2775.759537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2782.683525] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2783.843664] Lustre: 94669:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000008557cd25 x1875314972074240/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788439983 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2793.275237] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2800.724358] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2800.737659] Lustre: MGS: Client db604ed8-a018-4a26-aaae-86456da4b298 (at 0@lo) reconnecting [ 2800.775673] Lustre: 92816:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2800.806221] LustreError: 93532:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2800.816686] LustreError: 93532:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2800.838841] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2800.907572] Lustre: lustre-MDT0000: new disk, initializing [ 2801.042759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2801.079404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2801.099832] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2802.154273] Lustre: 95383:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000cb744203 x1875314972081536/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788440001 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2806.455288] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2807.268147] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2811.244093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2811.259627] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2811.361711] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2841.579450] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2841.588449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2841.606389] Lustre: Skipped 1 previous similar message [ 2841.618684] Lustre: Skipped 1 previous similar message [ 2844.715378] Lustre: server umount lustre-OST0000 complete [ 2851.810605] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2851.811632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2851.848590] Lustre: Skipped 1 previous similar message [ 2855.205682] Lustre: server umount lustre-MDT0000 complete [ 2856.944179] LustreError: 93518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2856.978471] LustreError: 93518:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2866.832167] Lustre: server umount lustre-MDT0001 complete [ 2870.926988] Lustre: server umount MGS complete [ 2882.151970] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 08:54:25 (1788440065) [ 2902.285483] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 2907.428726] Key type lgssc unregistered [ 2907.718976] LNet: 97885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2907.725969] LNetError: 97885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2907.736497] LNet: Removed LNI 192.168.201.146@tcp [ 2908.674563] Key type .llcrypt unregistered [ 2908.676435] Key type ._llcrypt unregistered [ 2925.523269] Key type ._llcrypt registered [ 2925.530752] Key type .llcrypt registered [ 2925.618904] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 2926.745777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2926.931373] alg: No test for adler32 (adler32-zlib) [ 2928.017485] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2928.387204] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 2930.120448] Key type lgssc registered [ 2931.619516] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2939.768824] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2939.793644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2941.172153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2946.508339] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2951.650964] LustreError: 99103:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2951.677301] LustreError: 99103:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2955.897918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2956.615103] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2961.765085] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2971.252760] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2977.784114] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2984.317142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2984.734901] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2985.802498] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2992.804147] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3001.311480] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3011.557490] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3011.559529] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3011.564399] LustreError: Skipped 1 previous similar message [ 3011.570807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3011.601530] Lustre: Skipped 1 previous similar message [ 3016.679259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3016.686786] Lustre: Skipped 1 previous similar message [ 3017.342924] Lustre: server umount lustre-OST0000 complete [ 3021.801967] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3021.809852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3021.818789] Lustre: Skipped 1 previous similar message [ 3021.825935] Lustre: Skipped 1 previous similar message [ 3026.930778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3028.027239] Lustre: server umount lustre-MDT0000 complete [ 3031.816455] LustreError: 99081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440215 with bad export cookie 12353899852452277361 [ 3031.838196] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3031.839452] LustreError: 99081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3032.163973] Lustre: server umount lustre-MDT0001 complete [ 3045.273656] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 3047.957691] Key type lgssc unregistered [ 3048.215229] LNet: 102634:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3048.226512] LNetError: 102634:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3048.243962] LNet: Removed LNI 192.168.201.146@tcp [ 3049.101350] Key type .llcrypt unregistered [ 3049.103741] Key type ._llcrypt unregistered [ 3066.520084] Key type ._llcrypt registered [ 3066.521173] Key type .llcrypt registered [ 3066.599596] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 3067.334774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3067.383402] alg: No test for adler32 (adler32-zlib) [ 3068.338564] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3068.666761] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 3070.344529] Key type lgssc registered [ 3072.398318] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3080.622657] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3080.645413] LustreError: 103708:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3080.684294] LustreError: 103715:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3080.691555] LustreError: 103708:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4976 [ 3080.699569] LustreError: 103708:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3080.723123] LustreError: 103708:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3080.727676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3082.563070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3091.175547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3091.677486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3096.730592] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3102.899368] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing lsmod [ 3105.541402] LustreError: 104923:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3105.579338] LustreError: 104924:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3105.585687] LustreError: 104923:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4963 [ 3105.595512] LustreError: 104923:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3105.600889] LustreError: 104923:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3105.607672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3106.193711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3107.263085] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3112.423881] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3112.426442] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3112.429426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3112.448751] Lustre: Skipped 2 previous similar messages [ 3115.658647] Lustre: server umount lustre-MDT0000 complete [ 3116.580776] LustreError: 103747:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3116.622729] LustreError: 103747:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3119.951760] LustreError: 103728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440303 with bad export cookie 1990229203424480204 [ 3119.973577] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3119.982465] LustreError: 103728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3120.737042] LustreError: 104185:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000014555e41 x1875315524654080/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3121.075976] Lustre: server umount lustre-MDT0001 complete [ 3135.173134] Lustre: server umount lustre-OST0000 complete [ 3145.268569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3146.020211] LustreError: 105988:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3146.116817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3151.330245] LustreError: 105989:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3152.507210] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3156.452472] LustreError: 105988:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3161.569800] LustreError: 105989:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3162.234474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3162.607892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3167.487224] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3176.241521] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3182.718130] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3190.302872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3190.753204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3191.836821] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3197.437385] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3205.956504] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3217.378147] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3217.408333] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3217.418946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3217.446794] Lustre: Skipped 3 previous similar messages [ 3220.933603] Lustre: server umount lustre-OST0000 complete [ 3227.618933] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3227.643067] Lustre: Skipped 2 previous similar messages [ 3227.648541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3227.666293] Lustre: Skipped 1 previous similar message [ 3231.645783] Lustre: server umount lustre-MDT0000 complete [ 3232.740227] LustreError: 107296:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3232.768469] LustreError: 107296:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3236.463893] LustreError: 105969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440419 with bad export cookie 1990229203424481128 [ 3236.485660] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3236.487319] LustreError: 105969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3236.762443] Lustre: server umount lustre-MDT0001 complete [ 3249.719658] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 3253.606234] Key type lgssc unregistered [ 3254.042976] LNet: 109524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3254.048576] LNetError: 109524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3254.075808] LNet: Removed LNI 192.168.201.146@tcp [ 3255.475115] Key type .llcrypt unregistered [ 3255.477354] Key type ._llcrypt unregistered [ 3282.560787] Key type ._llcrypt registered [ 3282.562742] Key type .llcrypt registered [ 3282.660849] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 3284.238135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3284.253795] alg: No test for adler32 (adler32-zlib) [ 3285.403573] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3285.581073] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 3287.219070] Key type lgssc registered [ 3288.564180] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3297.353216] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3297.372553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3298.898265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3303.607473] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3309.026056] LustreError: 110950:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3309.037628] LustreError: 110950:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3313.219868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3313.583578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3318.524157] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3326.086650] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3331.733972] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3338.369615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3338.645913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3343.801959] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3345.155413] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3353.086786] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3360.998342] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3364.876893] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3369.442346] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3369.448786] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3369.467407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3373.542153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3373.548647] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3378.662754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3378.679277] Lustre: Skipped 2 previous similar messages [ 3383.264228] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3383.623649] Lustre: server umount lustre-OST0000 complete [ 3388.907229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3388.912434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3388.926141] Lustre: Skipped 1 previous similar message [ 3388.949042] Lustre: Skipped 1 previous similar message [ 3393.976504] Lustre: server umount lustre-MDT0000 complete [ 3394.017220] LustreError: 111681:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3394.046446] LustreError: 111681:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3397.534597] LustreError: 110933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440580 with bad export cookie 8083114310169747981 [ 3397.549161] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3397.549775] LustreError: 110933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3397.976443] Lustre: server umount lustre-MDT0001 complete [ 3409.213645] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 3412.403857] Key type lgssc unregistered [ 3412.684070] LNet: 114589:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3412.692470] LNetError: 114589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3412.721947] LNet: Removed LNI 192.168.201.146@tcp [ 3413.796668] Key type .llcrypt unregistered [ 3413.799596] Key type ._llcrypt unregistered [ 3430.571645] Key type ._llcrypt registered [ 3430.577446] Key type .llcrypt registered [ 3430.687685] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 3431.852114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3431.892098] alg: No test for adler32 (adler32-zlib) [ 3432.901443] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3433.154149] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 3434.864722] Key type lgssc registered [ 3436.216660] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3443.558575] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3443.574491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3444.873955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3449.756889] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3455.462796] LustreError: 115811:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3455.478464] LustreError: 115811:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3457.926766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3458.416046] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3463.763471] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3471.198714] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3477.240387] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3484.029764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3484.500748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3490.698978] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3491.617510] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3499.328541] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3513.313630] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3513.324300] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3513.333558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3513.345697] Lustre: Skipped 1 previous similar message [ 3518.435940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3518.769955] Lustre: server umount lustre-OST0000 complete [ 3523.558090] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3523.563053] Lustre: Skipped 1 previous similar message [ 3523.568292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3523.575157] Lustre: Skipped 2 previous similar messages [ 3528.687209] LustreError: 115810:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3528.841889] Lustre: server umount lustre-MDT0000 complete [ 3532.491796] LustreError: 115790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440715 with bad export cookie 8700184670581205842 [ 3532.496387] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3532.516732] LustreError: 115790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3532.820290] Lustre: server umount lustre-MDT0001 complete [ 3545.760688] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 3548.645499] Key type lgssc unregistered [ 3548.999590] LNet: 119368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3549.008305] LNetError: 119368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3550.064567] LNet: Removed LNI 192.168.201.146@tcp [ 3551.054226] Key type .llcrypt unregistered [ 3551.056806] Key type ._llcrypt unregistered [ 3576.638547] Key type ._llcrypt registered [ 3576.641976] Key type .llcrypt registered [ 3576.788812] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 3577.931190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3577.978625] alg: No test for adler32 (adler32-zlib) [ 3579.186503] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3579.620139] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 3581.288121] Key type lgssc registered [ 3582.567584] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3590.611906] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3590.621777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3591.997275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3596.642965] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3602.410448] LustreError: 120782:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3602.429584] LustreError: 120782:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3604.618185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3604.922670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3610.105864] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3617.376377] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3624.400887] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3634.229449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3634.911101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3636.988523] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3642.631434] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3651.033659] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3658.086402] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3661.889876] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3661.894325] Lustre: Skipped 2 previous similar messages [ 3666.290994] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3666.297425] Lustre: Skipped 2 previous similar messages [ 3666.525151] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3666.600449] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3673.836507] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.46@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3673.848883] Lustre: Skipped 2 previous similar messages [ 3676.754427] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.46@tcp 192.168.0.[2,10]@tcp [ 3681.340532] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.46/24@tcp in log lustre-MDT0000 [ 3681.350452] Lustre: Skipped 2 previous similar messages [ 3686.985956] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.46/24@tcp [ 3686.994021] Lustre: Skipped 1 previous similar message [ 3693.537634] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3693.552897] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3693.584361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3696.616169] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3696.627991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3696.630941] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3698.810721] Lustre: server umount lustre-OST0000 complete [ 3706.851518] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3706.852986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3706.870863] Lustre: Skipped 1 previous similar message [ 3706.884393] Lustre: Skipped 2 previous similar messages [ 3709.174750] Lustre: server umount lustre-MDT0000 complete [ 3711.969957] LustreError: 120777:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3711.993340] LustreError: 120777:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3713.459082] LustreError: 120764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440896 with bad export cookie 4829616172968754650 [ 3713.461047] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3713.472378] LustreError: 120764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3714.078270] Lustre: server umount lustre-MDT0001 complete [ 3725.969478] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 3729.454660] Key type lgssc unregistered [ 3729.806033] LNet: 125524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3729.812889] LNetError: 125524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3730.857593] LNet: Removed LNI 192.168.201.146@tcp [ 3731.523846] Key type .llcrypt unregistered [ 3731.525264] Key type ._llcrypt unregistered [ 3767.627574] Key type ._llcrypt registered [ 3767.633300] Key type .llcrypt registered [ 3767.831134] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 3769.783347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3769.812171] alg: No test for adler32 (adler32-zlib) [ 3770.951749] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3771.458773] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 3773.200396] Key type lgssc registered [ 3774.719467] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3783.871365] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3783.900234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3785.621352] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3785.671470] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.46/24@tcp [ 3785.671757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3790.568690] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3794.918551] LustreError: 127079:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3794.940919] LustreError: 127079:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3800.038118] LustreError: 127080:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3800.380148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3800.690254] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3800.727235] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.46/24@tcp [ 3800.775754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3805.285835] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3813.064481] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3819.151462] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3825.119102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3825.461833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3830.618259] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3831.563686] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3839.859462] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3851.237666] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3851.247086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3851.250261] Lustre: Skipped 1 previous similar message [ 3851.251815] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3851.275220] Lustre: Skipped 1 previous similar message [ 3856.136936] Lustre: server umount lustre-OST0000 complete [ 3861.475217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3861.481180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3861.496603] Lustre: Skipped 1 previous similar message [ 3861.515733] Lustre: Skipped 1 previous similar message [ 3866.478588] Lustre: server umount lustre-MDT0000 complete [ 3866.596452] LustreError: 127080:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3866.622167] LustreError: 127080:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3871.315415] LustreError: 128929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441054 with bad export cookie 11695092710747601540 [ 3871.317733] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3871.332101] LustreError: 128929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3871.994980] Lustre: server umount lustre-MDT0001 complete [ 3886.242889] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 3890.786743] Key type lgssc unregistered [ 3891.182230] LNet: 130667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3891.186604] LNetError: 130667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3892.200724] LNet: Removed LNI 192.168.201.146@tcp [ 3893.249789] Key type .llcrypt unregistered [ 3893.252949] Key type ._llcrypt unregistered [ 3922.226394] Key type ._llcrypt registered [ 3922.228991] Key type .llcrypt registered [ 3922.360197] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 3938.161258] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 3939.510226] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3939.560981] alg: No test for adler32 (adler32-zlib) [ 3940.607213] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3940.826656] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 3942.488538] Key type lgssc registered [ 3943.772531] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3975.663912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3975.701198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3977.064271] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3977.102511] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3977.190815] Lustre: lustre-MDT0000: new disk, initializing [ 3977.320521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3977.335686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3981.209611] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3993.349777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3993.430859] Lustre: 134068:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3993.469931] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3993.476633] Lustre: Skipped 1 previous similar message [ 3993.564438] Lustre: lustre-MDT0001: new disk, initializing [ 3993.671731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3993.742953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3993.758707] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4000.362768] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4012.048696] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4019.435872] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4027.264560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4027.480752] Lustre: lustre-OST0000: new disk, initializing [ 4027.484760] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4027.489065] Lustre: 136040:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4027.595115] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4033.930660] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4035.688588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4035.701953] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4035.909027] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4045.525452] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4050.875662] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4051.158549] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4057.293106] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4057.557995] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4061.666645] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4061.676310] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4061.691790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4061.704402] Lustre: Skipped 1 previous similar message [ 4065.952549] Lustre: server umount lustre-OST0000 complete [ 4070.373930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4070.387966] LustreError: Skipped 1 previous similar message [ 4070.392203] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4070.402719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4070.410212] Lustre: Skipped 1 previous similar message [ 4071.906112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4071.910482] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4076.332122] Lustre: server umount lustre-MDT0000 complete [ 4079.954885] LustreError: 134058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441263 with bad export cookie 6650102027033770941 [ 4079.960842] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4079.964964] LustreError: 134058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4080.529218] Lustre: server umount lustre-MDT0001 complete [ 4089.969154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4090.865184] LustreError: 138257:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4090.904107] LustreError: 138257:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4090.965960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4095.971287] LustreError: 138258:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4096.088899] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4101.110169] LustreError: 138257:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4104.405664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4104.722249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4109.367826] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4117.430801] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4123.324664] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4130.504405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4130.946022] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4138.142054] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4146.576803] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4152.262741] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4152.529617] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4157.825711] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4158.087397] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4168.155687] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4168.346458] Lustre: lustre-OST0001: new disk, initializing [ 4168.350568] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4168.358484] Lustre: 141631:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4168.365115] Lustre: lustre-OST0001: Not available for connect from 192.168.201.46@tcp (not set up) [ 4168.370462] Lustre: Skipped 1 previous similar message [ 4168.491469] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4175.533171] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4178.006718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4178.018155] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4178.148840] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4184.916484] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4185.238391] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4190.167291] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4190.429273] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4196.851669] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff965a85b1f800.ost_server_uuid 50 [ 4198.632467] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff965a85b1f800.ost_server_uuid in IDLE state after 0 sec [ 4209.121588] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4209.124982] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4209.138595] LustreError: Skipped 1 previous similar message [ 4209.153822] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4209.165454] Lustre: Skipped 1 previous similar message [ 4214.543870] Lustre: server umount lustre-OST0001 complete [ 4219.374071] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4219.379820] LustreError: 141107:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4219.387310] Lustre: Skipped 1 previous similar message [ 4219.396085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4219.410240] LustreError: 141107:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4219.454910] Lustre: Skipped 4 previous similar messages [ 4224.488224] LustreError: 142934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4224.519835] LustreError: 142934:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4233.190455] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4233.566081] Lustre: server umount lustre-OST0000 complete [ 4237.798262] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4237.816074] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4237.850053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4237.852459] Lustre: Skipped 4 previous similar messages [ 4244.269710] Lustre: server umount lustre-MDT0000 complete [ 4244.975655] LustreError: 138257:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4245.001284] LustreError: 138257:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4248.455282] LustreError: 138238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441431 with bad export cookie 6650102027033771823 [ 4248.472434] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4248.481516] LustreError: 138238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4248.890238] Lustre: server umount lustre-MDT0001 complete [ 4269.579165] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 4271.917622] Key type lgssc unregistered [ 4272.181893] LNet: 144635:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4272.195168] LNetError: 144635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4273.257251] LNet: Removed LNI 192.168.201.146@tcp [ 4274.331213] Key type .llcrypt unregistered [ 4274.333203] Key type ._llcrypt unregistered [ 4323.104318] Key type ._llcrypt registered [ 4323.108700] Key type .llcrypt registered [ 4323.284377] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 4341.403878] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 4342.775931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4342.875063] alg: No test for adler32 (adler32-zlib) [ 4343.986164] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4344.296479] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 4345.976247] Key type lgssc registered [ 4347.478531] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4385.848912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4385.866351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4387.145748] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4387.203996] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4387.317216] Lustre: lustre-MDT0000: new disk, initializing [ 4387.412504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4387.432406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4392.947525] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4405.026972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4405.188561] Lustre: 148789:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4405.235527] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4405.239426] Lustre: Skipped 1 previous similar message [ 4405.336477] Lustre: lustre-MDT0001: new disk, initializing [ 4405.408140] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4405.455118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4405.487341] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4410.251493] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4421.071981] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4428.274957] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4436.260247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4436.660975] Lustre: lustre-OST0000: new disk, initializing [ 4436.668196] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4436.680938] Lustre: 150757:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4436.829700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4444.543388] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4446.244379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4446.256918] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4446.391900] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4454.537169] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4459.868971] Lustre: Failing over lustre-OST0000 [ 4460.204618] Lustre: server umount lustre-OST0000 complete [ 4478.432174] Lustre: 147010:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441645/real 1788441645] req@0000000095613c4e x1875316862650624/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788441661 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4478.438044] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4478.483029] Lustre: 147010:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4478.507872] Lustre: Skipped 1 previous similar message [ 4479.824362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4480.310530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4480.327616] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4483.408194] Lustre: 147013:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441650/real 1788441650] req@00000000854f80b7 x1875316862651520/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788441666 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4483.433175] Lustre: 147013:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4487.508183] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4488.352245] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4488.416328] Lustre: 147010:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441655/real 1788441655] req@0000000094dc4ce5 x1875316862652672/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788441671 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4488.450507] Lustre: 147010:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4488.463634] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4488.466313] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4491.994644] Lustre: Failing over lustre-MDT0000 [ 4492.424534] Lustre: server umount lustre-MDT0000 complete [ 4492.775147] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4492.781730] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4493.024541] Lustre: 147013:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441660/real 1788441660] req@00000000c5b3bd6b x1875316862653568/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788441676 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4493.057366] Lustre: 147013:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4493.799315] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4493.835944] Lustre: Skipped 1 previous similar message [ 4498.586378] LustreError: 148796:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4498.614302] LustreError: 148796:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4503.715417] LustreError: 148795:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4503.729308] LustreError: 148795:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4508.834782] LustreError: 148796:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4508.873151] LustreError: 148796:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4510.177094] Lustre: 147010:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441677/real 1788441677] req@00000000068d808d x1875316862663680/t0(0) o400->MGC192.168.201.146@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788441693 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4510.199777] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4511.646209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4513.943129] LustreError: 148794:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4513.990476] LustreError: 148794:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4519.063541] LustreError: 148796:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4519.086713] LustreError: 148796:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4520.419122] LustreError: 147009:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000be591440 x1875316862668928/t0(0) o250->MGC192.168.201.146@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4521.012854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4521.150970] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4524.211189] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4526.058877] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4526.070529] Lustre: Skipped 1 previous similar message [ 4526.095553] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4526.166827] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4527.891609] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4539.884934] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4539.888568] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4539.909342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4541.413385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4541.426203] Lustre: Skipped 1 previous similar message [ 4544.673882] Lustre: server umount lustre-OST0000 complete [ 4551.665096] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4551.673597] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4551.673978] Lustre: Skipped 1 previous similar message [ 4551.675870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4554.934532] Lustre: server umount lustre-MDT0000 complete [ 4556.774823] LustreError: 149969:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4556.804018] LustreError: 149969:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4559.589952] LustreError: 148781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441742 with bad export cookie 8722522196860595643 [ 4559.592465] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4559.598554] LustreError: 148781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4560.022579] Lustre: server umount lustre-MDT0001 complete [ 4571.687322] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 4574.572150] Key type lgssc unregistered [ 4574.793410] LNet: 154425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4574.801036] LNetError: 154425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4575.850875] LNet: Removed LNI 192.168.201.146@tcp [ 4576.926719] Key type .llcrypt unregistered [ 4576.931198] Key type ._llcrypt unregistered [ 4618.126310] Key type ._llcrypt registered [ 4618.129426] Key type .llcrypt registered [ 4618.272976] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 4620.218708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4620.249581] alg: No test for adler32 (adler32-zlib) [ 4621.438949] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4621.775749] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 4623.480642] Key type lgssc registered [ 4625.104490] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4633.332519] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4633.364683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4635.022217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4641.143618] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4645.351032] LustreError: 155943:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4645.379564] LustreError: 155943:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4650.468670] LustreError: 155944:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4653.298550] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4655.589437] LustreError: 155943:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4658.190027] LustreError: 155972:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4658.198889] LustreError: 155972:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4658.213691] LustreError: 155972:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 23, retries 0, failed: rc = -5 [ 4658.540463] Lustre: server umount lustre-MDT0000 complete [ 4674.872507] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 4686.320978] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 4724.595470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4724.891166] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4724.919776] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4725.006494] Lustre: lustre-MDT0000: new disk, initializing [ 4725.099714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4725.128079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4729.903884] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4741.605793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4741.761873] Lustre: 159278:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4741.830741] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4741.835711] Lustre: Skipped 1 previous similar message [ 4741.926873] Lustre: lustre-MDT0001: new disk, initializing [ 4742.073224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4742.111235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4742.118807] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4747.695045] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4758.615418] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4765.090987] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4773.360707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4773.763211] Lustre: lustre-OST0000: new disk, initializing [ 4773.771150] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4773.782606] Lustre: 161247:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4773.938962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4780.128766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4780.179111] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4780.435264] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4781.787229] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4791.722430] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4796.764916] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4798.599585] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 09:26:21 (1788441981) [ 5033.953434] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5033.959350] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5033.985276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5036.515023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5036.529850] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5036.550249] Lustre: Skipped 1 previous similar message [ 5041.635868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5041.638861] Lustre: Skipped 4 previous similar messages [ 5046.240134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5046.574729] Lustre: server umount lustre-MDT0000 complete [ 5046.756871] LustreError: 159288:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5050.941978] LustreError: 160072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442234 with bad export cookie 14918950424617763889 [ 5050.947687] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5050.962574] LustreError: 160072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5051.879228] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5051.881245] LustreError: 159285:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5051.899640] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5051.922274] LustreError: 159285:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5055.906607] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5057.394708] Lustre: server umount lustre-MDT0001 complete [ 5067.791186] Lustre: server umount lustre-OST0000 complete [ 5078.532368] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 5089.055752] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 5125.084036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5125.245389] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5125.260799] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5125.319262] Lustre: lustre-MDT0000: new disk, initializing [ 5125.376974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5125.391467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5129.619361] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5141.172467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5141.234808] Lustre: 165907:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5141.286834] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5141.291142] Lustre: Skipped 1 previous similar message [ 5141.387781] Lustre: lustre-MDT0001: new disk, initializing [ 5141.526488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5141.570090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5141.591545] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5146.640083] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5157.957942] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5165.051772] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5173.115708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5173.517246] Lustre: lustre-OST0000: new disk, initializing [ 5173.523685] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5173.532665] Lustre: 167879:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5173.683525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5180.908452] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5182.508072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5182.511708] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5182.592529] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5191.407558] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5196.550906] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5196.800647] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5201.392200] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5201.641211] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5208.035585] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5208.046118] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5208.058599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5218.272437] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5218.516626] Lustre: server umount lustre-OST0000 complete [ 5223.396054] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5223.408979] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5223.424254] Lustre: Skipped 1 previous similar message [ 5228.895397] Lustre: server umount lustre-MDT0000 complete [ 5229.038784] LustreError: 165919:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5229.050928] LustreError: 165919:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5232.338263] LustreError: 165900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442415 with bad export cookie 14918950424617922719 [ 5232.352393] LustreError: 165900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5232.352754] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5232.867445] Lustre: server umount lustre-MDT0001 complete [ 5240.528155] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:33:43 (1788442423) [ 5256.344848] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 5266.764811] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 5304.395591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5304.617797] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5304.640248] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5304.718744] Lustre: lustre-MDT0000: new disk, initializing [ 5304.819506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5304.832515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5308.912588] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5319.210491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5319.299395] Lustre: 172635:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5319.524524] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5324.298896] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5334.175408] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5339.940383] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5346.200371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5346.501390] Lustre: lustre-OST0000: new disk, initializing [ 5346.508501] Lustre: Skipped 1 previous similar message [ 5346.517603] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5346.522165] Lustre: Skipped 2 previous similar messages [ 5346.528575] Lustre: 174613:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5352.856981] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5353.523496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5353.544083] Lustre: Skipped 1 previous similar message [ 5353.554180] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5353.704838] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5361.921793] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5369.316470] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5369.319424] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5369.323175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5369.330898] Lustre: Skipped 2 previous similar messages [ 5369.350441] Lustre: Skipped 6 previous similar messages [ 5375.440476] Lustre: server umount lustre-OST0000 complete [ 5385.662113] Lustre: server umount lustre-MDT0000 complete [ 5389.344424] LustreError: 172627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442572 with bad export cookie 14918950424617923608 [ 5389.350795] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5389.358495] LustreError: 172627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5389.660723] Lustre: server umount lustre-MDT0001 complete [ 5398.197311] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:36:20 (1788442580) [ 5414.597848] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 5424.775992] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 5458.954776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5459.166234] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5459.193469] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5459.262312] Lustre: lustre-MDT0000: new disk, initializing [ 5459.374399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5459.381889] Lustre: Skipped 2 previous similar messages [ 5459.398864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5464.089405] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5474.208902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5474.461349] Lustre: 178937:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5474.577269] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5474.585507] Lustre: Skipped 6 previous similar messages [ 5474.881189] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5479.052628] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5490.267834] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5498.688076] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5508.035579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5508.398067] Lustre: 180914:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5517.155893] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5517.916669] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5518.071130] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5527.993612] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5538.786992] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5538.788654] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5538.798726] LustreError: Skipped 1 previous similar message [ 5538.801177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5538.831985] Lustre: Skipped 3 previous similar messages [ 5541.546724] Lustre: server umount lustre-OST0000 complete [ 5551.640465] Lustre: server umount lustre-MDT0000 complete [ 5554.145607] LustreError: 181739:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5554.155840] LustreError: 181739:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5555.289232] LustreError: 178929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442738 with bad export cookie 14918950424617924693 [ 5555.302018] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5555.590966] Lustre: server umount lustre-MDT0001 complete [ 5568.337579] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 5571.323496] Key type lgssc unregistered [ 5571.624447] LNet: 182771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5571.630835] LNetError: 182771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5571.647418] LNet: Removed LNI 192.168.201.146@tcp [ 5572.733943] Key type .llcrypt unregistered [ 5572.736994] Key type ._llcrypt unregistered [ 5594.976686] Key type ._llcrypt registered [ 5594.980418] Key type .llcrypt registered [ 5595.071931] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 5596.230261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5596.240069] alg: No test for adler32 (adler32-zlib) [ 5597.530866] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5597.876432] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 5599.536167] Key type lgssc registered [ 5601.125372] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5614.310575] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5614.345661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5615.910727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5622.390069] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5626.337890] LustreError: 184181:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5626.379094] LustreError: 184181:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5630.829536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5631.318451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5636.946496] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5646.778986] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5653.618707] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5661.234832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5661.807097] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5669.331673] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5678.263126] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5686.419291] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5712.359054] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5712.359640] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5712.376087] LustreError: Skipped 1 previous similar message [ 5712.405222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5712.426313] Lustre: Skipped 1 previous similar message [ 5713.873321] Lustre: server umount lustre-OST0000 complete [ 5718.502216] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5718.522752] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5718.533735] Lustre: Skipped 1 previous similar message [ 5718.543072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5722.594841] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5722.597555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5722.667804] Lustre: Skipped 1 previous similar message [ 5724.678889] Lustre: server umount lustre-MDT0000 complete [ 5727.713721] LustreError: 186039:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5727.733712] LustreError: 186039:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5729.002147] LustreError: 184163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442912 with bad export cookie 4502036147088704030 [ 5729.013827] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5729.015965] LustreError: 184163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5729.467425] Lustre: server umount lustre-MDT0001 complete [ 5740.449737] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 5742.897451] Key type lgssc unregistered [ 5743.184624] LNet: 187741:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5743.189106] LNetError: 187741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5744.249339] LNet: Removed LNI 192.168.201.146@tcp [ 5745.160304] Key type .llcrypt unregistered [ 5745.162741] Key type ._llcrypt unregistered [ 5771.127094] Key type ._llcrypt registered [ 5771.132454] Key type .llcrypt registered [ 5771.251166] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 5773.086122] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5773.123858] alg: No test for adler32 (adler32-zlib) [ 5774.246277] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5774.582984] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 5776.288691] Key type lgssc registered [ 5777.912065] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5786.595967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5786.629285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5788.148157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5792.885751] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5798.378270] LustreError: 189156:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5798.428828] LustreError: 189156:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5802.994381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5803.257436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5809.549189] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5819.466878] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5825.864801] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5832.002608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5832.386470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5834.436704] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5840.267791] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5848.652983] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5856.958103] Lustre: lustre-OST0000: Not available for connect from 192.168.201.46@tcp (stopping) [ 5859.299822] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5859.306715] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5859.319958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5859.326663] Lustre: Skipped 1 previous similar message [ 5862.041467] Lustre: lustre-OST0000: Not available for connect from 192.168.201.46@tcp (stopping) [ 5864.421388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5864.436751] Lustre: Skipped 1 previous similar message [ 5869.561801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5869.576689] Lustre: Skipped 2 previous similar messages [ 5871.073177] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5871.500569] Lustre: server umount lustre-OST0000 complete [ 5879.343153] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff965a8812a800.ost_server_uuid 50 [ 5881.737712] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5896.285034] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5896.289868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5896.325217] Lustre: Skipped 2 previous similar messages [ 5896.341620] Lustre: Skipped 1 previous similar message [ 5902.603432] Lustre: server umount lustre-MDT0000 complete [ 5904.865357] LustreError: 190097:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5904.895791] LustreError: 190097:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5906.484730] LustreError: 189137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443089 with bad export cookie 5217748275944992814 [ 5906.492615] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5906.494988] LustreError: 189137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5906.839133] Lustre: server umount lustre-MDT0001 complete [ 5915.376601] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 09:44:58 (1788443098) [ 5922.315832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5922.896562] LustreError: 193077:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5922.976772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5928.102681] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5928.424225] LustreError: 193078:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5933.546128] LustreError: 193077:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5937.668464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5938.193237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5944.061199] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5953.266664] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5960.633466] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5969.531527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5969.958279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5977.634205] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5986.971370] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5993.541578] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5995.788395] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5996.054378] Lustre: lustre-OST0001: new disk, initializing [ 5996.065667] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5996.070118] Lustre: 195921:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5996.190760] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6004.438440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6004.458707] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6004.707124] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6004.799269] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6015.200439] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6019.049064] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6019.058160] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6019.068903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6019.077702] Lustre: Skipped 2 previous similar messages [ 6020.075663] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6032.352428] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6032.438229] Lustre: server umount lustre-OST0000 complete [ 6035.442600] LustreError: 196269:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6035.489670] LustreError: 196269:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6040.232326] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6040.472296] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6046.854677] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6047.212809] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6051.488421] LustreError: 194903:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6051.517615] LustreError: 194903:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 6066.150865] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6066.155160] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6066.174862] Lustre: Skipped 1 previous similar message [ 6066.181755] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6066.189128] Lustre: Skipped 10 previous similar messages [ 6067.517483] Lustre: server umount lustre-OST0001 complete [ 6076.387113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6076.398951] Lustre: Skipped 1 previous similar message [ 6079.842249] Lustre: server umount lustre-MDT0000 complete [ 6083.284882] LustreError: 193059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443266 with bad export cookie 5217748275944993885 [ 6083.289536] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6083.301592] LustreError: 193059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6083.601365] Lustre: server umount lustre-MDT0001 complete [ 6108.229576] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 09:48:11 (1788443291) [ 6114.662283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6114.991217] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6115.078643] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6115.358419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6119.977333] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6128.737853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6128.817259] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6128.844290] Lustre: 199367:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6133.571448] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6141.675927] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6148.424942] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6156.762593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6156.920293] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6164.582989] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6165.137922] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6172.465068] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6179.480647] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6179.546168] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6179.698484] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6179.704316] Lustre: Skipped 2 previous similar messages [ 6184.911308] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6191.898366] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6195.685255] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6195.693241] LustreError: Skipped 1 previous similar message [ 6195.701887] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6195.714750] Lustre: Skipped 1 previous similar message [ 6195.727087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6195.733869] Lustre: Skipped 2 previous similar messages [ 6201.168860] Lustre: server umount lustre-OST0000 complete [ 6202.527360] LustreError: 201219:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6202.560676] LustreError: 201219:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 6218.211896] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6218.217958] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6218.219200] LustreError: Skipped 2 previous similar messages [ 6218.228170] Lustre: Skipped 2 previous similar messages [ 6220.276863] Lustre: server umount lustre-OST0001 complete [ 6231.700912] Lustre: server umount lustre-MDT0000 complete [ 6237.103939] LustreError: 199357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443420 with bad export cookie 5217748275944995292 [ 6237.106086] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6237.125797] LustreError: 199357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6238.130451] Lustre: server umount lustre-MDT0001 complete [ 6269.677777] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 09:50:52 (1788443452) [ 6288.350314] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 6300.551683] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 6341.157552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6341.543429] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6341.597560] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6341.712874] Lustre: lustre-MDT0000: new disk, initializing [ 6341.812105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6341.860737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6346.449427] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6356.493543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6356.664984] Lustre: 207466:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6356.703826] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6356.707516] Lustre: Skipped 1 previous similar message [ 6356.760701] Lustre: lustre-MDT0001: new disk, initializing [ 6356.878769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6356.895142] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6362.560111] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6375.482203] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6382.208514] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6388.736605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6389.098967] Lustre: lustre-OST0000: new disk, initializing [ 6389.104144] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6389.112685] Lustre: 209435:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6395.522275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6395.545444] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6395.796121] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6396.630523] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6406.893618] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6413.190325] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6413.544170] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6420.056935] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6420.572679] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6426.603790] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6426.611361] LustreError: Skipped 1 previous similar message [ 6426.612582] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6426.637943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6426.650085] Lustre: Skipped 3 previous similar messages [ 6426.665663] Lustre: Skipped 10 previous similar messages [ 6429.388548] Lustre: server umount lustre-OST0000 complete [ 6440.945732] Lustre: server umount lustre-MDT0000 complete [ 6441.954846] LustreError: 207478:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6441.964498] LustreError: 207478:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 6445.819623] LustreError: 207457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443629 with bad export cookie 5217748275944996853 [ 6445.843287] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6446.578438] Lustre: server umount lustre-MDT0001 complete [ 6456.948775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6463.641904] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6472.351627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6478.891406] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6487.754349] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6494.131370] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6501.007554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6507.469574] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6516.311428] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6523.273689] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6523.689467] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6530.045390] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6530.247315] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6533.094953] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6533.101792] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6533.109211] Lustre: Skipped 3 previous similar messages [ 6546.401214] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6546.585393] Lustre: server umount lustre-OST0000 complete [ 6552.810912] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6557.905952] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6587.881092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6595.756382] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6598.452590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6598.461377] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6598.466270] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6598.477611] Lustre: Skipped 1 previous similar message [ 6604.934943] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6606.264646] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6614.000080] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6616.062338] Lustre: server umount lustre-OST0000 complete [ 6627.018577] Lustre: server umount lustre-MDT0000 complete [ 6629.351563] LustreError: 211650:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6629.370175] LustreError: 211650:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6630.745834] LustreError: 211633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443814 with bad export cookie 5217748275944997735 [ 6630.751253] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6630.757377] LustreError: 211633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6631.091728] Lustre: server umount lustre-MDT0001 complete [ 6639.882170] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 09:57:02 (1788443822) [ 6646.581350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6647.172526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6647.179241] Lustre: Skipped 6 previous similar messages [ 6652.009944] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6660.537131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6666.328922] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6673.101459] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6678.959690] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6685.974672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6693.074980] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6699.886930] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6704.482180] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6704.665833] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6709.844065] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6710.045310] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6716.775159] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6717.028628] Lustre: lustre-OST0001: new disk, initializing [ 6717.031928] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6717.037172] Lustre: 220993:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6723.472889] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6725.211538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6725.222502] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6725.281470] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6732.567426] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6737.345775] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6737.519711] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6742.369619] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6742.675146] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6746.097107] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6746.109786] LustreError: Skipped 2 previous similar messages [ 6746.116435] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6746.130669] Lustre: Skipped 4 previous similar messages [ 6746.140481] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6746.144662] Lustre: Skipped 14 previous similar messages [ 6750.521918] Lustre: server umount lustre-OST0001 complete [ 6756.587790] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6756.810524] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6762.263168] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6762.544400] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6792.414527] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6797.844711] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6797.880546] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6797.881440] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6798.770202] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6805.742879] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6807.604279] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6815.907279] Lustre: server umount lustre-OST0001 complete [ 6839.895729] Lustre: server umount lustre-OST0000 complete [ 6859.232360] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6859.522614] Lustre: server umount lustre-MDT0000 complete [ 6864.090697] LustreError: 219469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444047 with bad export cookie 5217748275944998694 [ 6864.101297] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6864.102748] LustreError: 219469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6870.896310] Lustre: server umount lustre-MDT0001 complete [ 6901.104950] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 10:01:23 (1788444083) [ 6909.434551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6909.660486] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6909.696348] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6914.152838] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6923.157962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6923.324274] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6923.372653] Lustre: 226258:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6927.747160] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6936.439716] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6942.265070] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6949.006140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6949.203182] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6956.270795] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6965.102608] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6975.307291] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6975.356195] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6975.457398] LustreError: 228104:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6975.468270] LustreError: 228104:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 37 previous similar messages [ 6982.006485] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6989.004223] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6991.146491] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6996.215345] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6996.604569] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7002.215648] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7002.562387] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7009.790587] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff965aa00cb800.ost_server_uuid 50 [ 7011.532956] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff965aa00cb800.ost_server_uuid in IDLE state after 0 sec [ 7012.907328] Lustre: Permanently deactivating lustre-OST0001 [ 7012.974975] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7017.065533] Lustre: Permanently reactivating lustre-OST0001 [ 7017.073399] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7017.079648] Lustre: Skipped 2 previous similar messages [ 7022.560915] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7022.563285] LustreError: Skipped 4 previous similar messages [ 7022.566546] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7022.591233] Lustre: Skipped 7 previous similar messages [ 7034.336481] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7034.422148] Lustre: server umount lustre-OST0001 complete [ 7044.000760] Lustre: server umount lustre-OST0000 complete [ 7054.722590] Lustre: server umount lustre-MDT0000 complete [ 7060.308493] LustreError: 226249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444243 with bad export cookie 5217748275944999989 [ 7060.323090] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7060.328017] LustreError: 226249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7060.780821] Lustre: server umount lustre-MDT0001 complete [ 7085.897338] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 10:04:28 (1788444268) [ 7094.275229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7094.529959] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7094.555750] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7094.559983] Lustre: Skipped 2 previous similar messages [ 7098.261921] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7105.654865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7105.770573] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7105.819118] Lustre: 232959:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7112.393527] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7123.274029] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7133.169290] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7140.440411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7140.714385] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7140.735555] Lustre: Permanently deactivating lustre-OST0000 [ 7140.752132] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7147.410743] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7150.660461] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7150.752593] LustreError: 235355:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -5 [ 7150.764599] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7153.336336] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7159.746495] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7159.788580] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7159.938236] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7159.949692] Lustre: Skipped 11 previous similar messages [ 7166.147917] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7167.176084] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7173.441156] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7175.322157] Lustre: Permanently reactivating lustre-OST0000 [ 7175.345388] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7175.348649] Lustre: Skipped 2 previous similar messages [ 7180.904169] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7180.916412] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7180.945050] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7180.976576] Lustre: Skipped 1 previous similar message [ 7181.029521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7181.058038] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7198.677760] Lustre: server umount lustre-OST0001 complete [ 7216.096131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7216.408260] Lustre: server umount lustre-OST0000 complete [ 7227.363481] Lustre: server umount lustre-MDT0000 complete [ 7230.639398] LustreError: 232949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444413 with bad export cookie 5217748275945001648 [ 7230.650375] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7230.654251] LustreError: 232949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7230.865418] Lustre: server umount lustre-MDT0001 complete [ 7241.541699] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 7246.444850] Key type lgssc unregistered [ 7246.897704] LNet: 237840:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7246.901600] LNetError: 237840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7246.939769] LNet: Removed LNI 192.168.201.146@tcp [ 7248.503634] Key type .llcrypt unregistered [ 7248.529248] Key type ._llcrypt unregistered [ 7273.629183] Key type ._llcrypt registered [ 7273.631754] Key type .llcrypt registered [ 7273.812810] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 7275.190384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7275.226254] alg: No test for adler32 (adler32-zlib) [ 7276.561676] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7276.880350] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 7278.594618] Key type lgssc registered [ 7279.895679] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7291.248725] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7291.286093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7292.714570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7297.681112] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7302.129252] LustreError: 239305:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7302.139906] LustreError: 239305:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7306.931793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7307.145770] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7307.187341] Lustre: 239294:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7307.200163] Lustre: Permanently deactivating lustre-MDT0001 [ 7307.220618] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7307.246428] LustreError: 239304:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7307.516324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7312.249468] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7313.938437] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7319.560232] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7325.775580] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7332.982221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7333.376231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7337.463355] LustreError: 241147:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7337.475784] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7340.319909] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7342.566470] LustreError: 241147:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7347.690831] LustreError: 241145:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7348.938086] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7351.808059] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 7356.907865] LustreError: 241368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7356.922692] LustreError: 241368:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7357.341335] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7357.660544] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7363.069333] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7363.073251] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7365.674084] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7373.932817] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7376.673606] Lustre: Permanently reactivating lustre-MDT0001 [ 7376.686754] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7376.689916] Lustre: Skipped 2 previous similar messages [ 7379.482382] Lustre: lustre-MDT0001: Client 20cea4d7-b61d-4444-9c52-92d0ace56240 (at 192.168.201.46@tcp) reconnecting [ 7382.605821] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7382.616035] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7382.631900] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7382.650852] LustreError: 242974:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7382.651148] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7382.661426] LustreError: 242974:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7382.682484] LustreError: 242974:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.146@tcp: cfg command failed: rc = -108 [ 7382.690823] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7382.690823] [ 7382.697306] LustreError: 239296:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7385.392444] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7386.750987] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7392.805445] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7393.011883] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7398.664709] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7399.029137] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7401.740715] Lustre: Permanently deactivating lustre-MDT0001 [ 7401.760456] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7401.764830] Lustre: Skipped 2 previous similar messages [ 7408.841358] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7418.342567] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7418.345186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7418.364581] Lustre: Skipped 1 previous similar message [ 7418.384227] Lustre: Skipped 3 previous similar messages [ 7419.553371] Lustre: server umount lustre-MDT0000 complete [ 7423.221842] LustreError: 240043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444606 with bad export cookie 4008450989051202006 [ 7423.232562] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7423.235324] LustreError: 240043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7423.465358] LustreError: 241909:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7423.483857] LustreError: 241909:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7423.664503] Lustre: server umount lustre-MDT0001 complete [ 7429.192967] Lustre: server umount lustre-OST0000 complete [ 7442.203759] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 10:10:25 (1788444625) [ 7452.360123] Lustre: server umount lustre-OST0001 complete [ 7460.273210] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 7471.108936] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 7510.656306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7511.037632] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7511.058232] Lustre: Skipped 2 previous similar messages [ 7511.090343] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7511.225792] Lustre: lustre-MDT0000: new disk, initializing [ 7511.380459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7511.416051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7517.058838] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7527.599091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7527.711196] Lustre: 247590:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7527.767744] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7527.779521] Lustre: Skipped 1 previous similar message [ 7527.897672] Lustre: lustre-MDT0001: new disk, initializing [ 7527.975448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7528.015292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7528.030060] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7534.710374] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7548.447648] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7556.067763] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7566.086432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7566.376266] Lustre: lustre-OST0000: new disk, initializing [ 7566.380027] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7566.385067] Lustre: 249560:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7566.443379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7573.073934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7573.082851] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7573.309616] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7573.688613] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7584.907367] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7586.964240] LustreError: 247597:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7596.634189] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7596.974595] Lustre: lustre-OST0001: new disk, initializing [ 7596.977682] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7596.984284] Lustre: 250675:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7597.010379] LustreError: 247597:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7597.026686] LustreError: 250328:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7603.285440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7603.296215] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7603.398369] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7604.900243] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7607.080255] LustreError: 250328:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7615.280755] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7617.172578] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7619.052218] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7619.054206] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7619.057562] Lustre: Skipped 2 previous similar messages [ 7619.058942] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7619.063719] Lustre: Skipped 1 previous similar message [ 7624.181368] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7624.191586] Lustre: Skipped 1 previous similar message [ 7629.329377] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7633.376197] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7633.642871] Lustre: server umount lustre-OST0001 complete [ 7634.418504] LustreError: 249545:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7634.435737] LustreError: 249545:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7654.889181] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7654.901052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7654.913880] Lustre: Skipped 2 previous similar messages [ 7654.926298] Lustre: Skipped 2 previous similar messages [ 7664.037803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7664.057238] Lustre: Skipped 3 previous similar messages [ 7666.144152] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7666.333961] Lustre: server umount lustre-OST0000 complete [ 7671.267724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7671.277385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7676.667473] Lustre: server umount lustre-MDT0000 complete [ 7681.387342] LustreError: 247581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444864 with bad export cookie 4008450989051205044 [ 7681.414454] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7688.045688] Lustre: server umount lustre-MDT0001 complete [ 7703.773518] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 7706.460329] Key type lgssc unregistered [ 7706.780972] LNet: 252710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7706.789129] LNetError: 252710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7706.803375] LNet: Removed LNI 192.168.201.146@tcp [ 7707.724780] Key type .llcrypt unregistered [ 7707.727089] Key type ._llcrypt unregistered [ 7755.013519] Key type ._llcrypt registered [ 7755.019187] Key type .llcrypt registered [ 7755.123542] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 7756.373747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7756.424620] alg: No test for adler32 (adler32-zlib) [ 7757.543926] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7757.740111] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 7759.408192] Key type lgssc registered [ 7761.161620] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7770.261691] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7770.279765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7771.740948] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7771.765281] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7771.963243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7777.794343] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7788.879811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7788.954957] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7788.985248] Lustre: 254920:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7789.327653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7794.428719] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7803.737576] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7810.377180] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7818.768982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7818.998832] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7819.193149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7826.269419] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7829.275795] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7835.497623] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7890.923619] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7890.925395] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7890.937454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7890.938266] LustreError: Skipped 1 previous similar message [ 7890.983436] Lustre: Skipped 1 previous similar message [ 7894.627445] Lustre: server umount lustre-OST0000 complete [ 7899.398542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7913.291362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7913.908046] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7922.160291] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7931.332575] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7931.364143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7931.369470] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7931.793061] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7972.322350] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7972.322765] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7972.335257] LustreError: Skipped 1 previous similar message [ 7972.378503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7972.392744] Lustre: Skipped 2 previous similar messages [ 7977.449221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7977.456416] Lustre: Skipped 1 previous similar message [ 7977.940694] Lustre: server umount lustre-OST0000 complete [ 7982.565839] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7982.576518] Lustre: Skipped 1 previous similar message [ 7982.584369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7987.681720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7987.689285] Lustre: Skipped 2 previous similar messages [ 7996.384290] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7996.588507] Lustre: server umount lustre-MDT0000 complete [ 8001.120995] LustreError: 254911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445184 with bad export cookie 14097750665439439374 [ 8001.126434] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8001.130827] LustreError: 254911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8001.791496] Lustre: server umount lustre-MDT0001 complete [ 8014.797955] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 8017.948221] Key type lgssc unregistered [ 8018.312130] LNet: 260346:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8018.340743] LNetError: 260346:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8018.367167] LNet: Removed LNI 192.168.201.146@tcp [ 8019.495169] Key type .llcrypt unregistered [ 8019.497406] Key type ._llcrypt unregistered [ 8046.210781] Key type ._llcrypt registered [ 8046.212954] Key type .llcrypt registered [ 8046.296053] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 8047.775610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8047.793421] alg: No test for adler32 (adler32-zlib) [ 8048.937862] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8049.224235] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 8050.880184] Key type lgssc registered [ 8052.096971] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8061.517992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8061.544356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8063.242324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8069.094578] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 8073.710815] LustreError: 261759:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8073.736805] LustreError: 261759:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8078.818401] LustreError: 261758:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8079.210484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8079.625276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8085.454779] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 8094.596225] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8101.689961] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8109.767769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8110.018755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8119.337351] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 8128.983834] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8133.557213] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8159.202779] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8159.213653] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8159.216564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8159.226230] Lustre: Skipped 1 previous similar message [ 8159.236637] Lustre: Skipped 1 previous similar message [ 8164.327630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8164.336383] Lustre: Skipped 1 previous similar message [ 8164.617730] Lustre: server umount lustre-OST0000 complete [ 8169.442454] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8169.444094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8169.446615] Lustre: Skipped 1 previous similar message [ 8174.232662] Lustre: server umount lustre-MDT0000 complete [ 8174.562399] LustreError: 261754:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8174.580190] LustreError: 261754:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8178.449383] LustreError: 261741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445361 with bad export cookie 3945155696231668042 [ 8178.467530] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8178.468552] LustreError: 261741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8178.776912] Lustre: server umount lustre-MDT0001 complete [ 8190.878405] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 8194.114248] Key type lgssc unregistered [ 8194.596093] LNet: 266068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8194.612542] LNetError: 266068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8194.633985] LNet: Removed LNI 192.168.201.146@tcp [ 8195.501159] Key type .llcrypt unregistered [ 8195.502933] Key type ._llcrypt unregistered [ 8217.110358] Key type ._llcrypt registered [ 8217.112745] Key type .llcrypt registered [ 8217.276775] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 8218.975885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8219.099162] alg: No test for adler32 (adler32-zlib) [ 8220.607803] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8220.978609] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 8222.706863] Key type lgssc registered [ 8224.695696] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8233.499162] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8233.515681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8234.831695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8239.732828] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 8245.219505] LustreError: 267289:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8245.251392] LustreError: 267289:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8248.855737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8249.348815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8254.321124] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 8261.345557] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8267.703747] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8275.798884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8276.177333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8280.278978] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8284.888242] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 8291.607611] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8306.145719] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8306.158247] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8306.176251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8310.242858] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8310.246771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8310.256676] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8311.988640] Lustre: server umount lustre-OST0000 complete [ 8315.873833] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8315.888299] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8315.906773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8315.919268] Lustre: Skipped 1 previous similar message [ 8320.481294] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8320.489574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8320.504465] Lustre: Skipped 1 previous similar message [ 8325.603812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8330.209648] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8330.346550] Lustre: server umount lustre-MDT0000 complete [ 8330.731281] LustreError: 267286:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8330.757035] LustreError: 267286:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8333.985388] LustreError: 267270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445517 with bad export cookie 10931318004610593717 [ 8334.009624] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8334.307853] Lustre: server umount lustre-MDT0001 complete [ 8345.072652] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 8347.392899] Key type lgssc unregistered [ 8347.670883] LNet: 270968:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8347.678736] LNetError: 270968:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8347.706067] LNet: Removed LNI 192.168.201.146@tcp [ 8348.580338] Key type .llcrypt unregistered [ 8348.582730] Key type ._llcrypt unregistered [ 8371.867096] Key type ._llcrypt registered [ 8371.869677] Key type .llcrypt registered [ 8372.034314] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 8373.166559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8373.191158] alg: No test for adler32 (adler32-zlib) [ 8374.519811] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8374.833953] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 8376.528200] Key type lgssc registered [ 8377.323941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8383.891062] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8383.942224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8385.728072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8392.009690] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 8395.491150] LustreError: 272378:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8395.530902] LustreError: 272378:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8400.871308] LustreError: 272379:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8405.987759] LustreError: 272378:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8406.432090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8407.089617] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8413.376192] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 8423.322413] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8429.450419] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8436.438692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8436.921306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8443.337253] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 8450.524554] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8455.358190] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8481.249235] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8481.261658] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8481.267941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8481.294392] Lustre: Skipped 1 previous similar message [ 8485.800554] Lustre: server umount lustre-OST0000 complete [ 8491.492164] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8491.512721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8491.525545] Lustre: Skipped 2 previous similar messages [ 8496.610094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8496.625954] Lustre: Skipped 1 previous similar message [ 8501.734893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8501.744203] Lustre: Skipped 1 previous similar message [ 8504.288142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8504.459083] Lustre: server umount lustre-MDT0000 complete [ 8506.849669] LustreError: 272373:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8506.866649] LustreError: 272373:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8507.843814] LustreError: 275753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445691 with bad export cookie 2340306639893721136 [ 8507.859844] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8507.860630] LustreError: 275753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8508.132826] Lustre: server umount lustre-MDT0001 complete [ 8518.730410] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 8522.320328] Key type lgssc unregistered [ 8522.740693] LNet: 276709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8522.747408] LNetError: 276709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8522.766039] LNet: Removed LNI 192.168.201.146@tcp [ 8524.048409] Key type .llcrypt unregistered [ 8524.050172] Key type ._llcrypt unregistered [ 8547.355463] Key type ._llcrypt registered [ 8547.357718] Key type .llcrypt registered [ 8547.521277] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 8549.053248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8549.103787] alg: No test for adler32 (adler32-zlib) [ 8550.423972] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8550.740259] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 8552.512173] Key type lgssc registered [ 8553.850391] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8563.552883] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8563.575782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8565.364922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8569.896561] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 8574.949574] LustreError: 277946:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8574.971670] LustreError: 277946:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8578.292574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8578.751485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8584.286964] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 8591.936392] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8598.456713] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8607.137722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8607.656878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8610.750586] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8614.480993] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 8622.442845] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8633.322673] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8633.324976] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8633.337690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8633.351433] Lustre: Skipped 1 previous similar message [ 8633.381774] Lustre: Skipped 1 previous similar message [ 8635.115752] Lustre: server umount lustre-OST0000 complete [ 8640.481774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8640.485558] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8640.493358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8643.557553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8643.561554] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8645.826205] Lustre: server umount lustre-MDT0000 complete [ 8648.674418] LustreError: 277934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8648.705533] LustreError: 277934:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8649.547610] LustreError: 278708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445832 with bad export cookie 11505622574982097399 [ 8649.550400] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8649.555960] LustreError: 278708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8649.857386] Lustre: server umount lustre-MDT0001 complete [ 8662.874857] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 8666.603842] Key type lgssc unregistered [ 8667.004029] LNet: 281646:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8667.012189] LNetError: 281646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8667.033715] LNet: Removed LNI 192.168.201.146@tcp [ 8668.011397] Key type .llcrypt unregistered [ 8668.013918] Key type ._llcrypt unregistered [ 8680.479750] Key type ._llcrypt registered [ 8680.490161] Key type .llcrypt registered [ 8680.623092] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8699.065529] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 8714.820640] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 8715.791408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8715.859558] alg: No test for adler32 (adler32-zlib) [ 8716.969552] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8717.260224] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 8718.984180] Key type lgssc registered [ 8720.238466] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8760.092643] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8760.119415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8761.392787] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8761.424484] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8761.514874] Lustre: lustre-MDT0000: new disk, initializing [ 8761.572520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8761.590468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8766.516607] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 8777.236621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8777.426799] Lustre: 285198:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8777.483444] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8777.487445] Lustre: Skipped 1 previous similar message [ 8777.586989] Lustre: lustre-MDT0001: new disk, initializing [ 8777.698144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8777.736676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8777.746529] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8782.544812] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 8793.461788] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8799.210483] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8807.082359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8807.401873] Lustre: lustre-OST0000: new disk, initializing [ 8807.405781] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8807.411084] Lustre: 287167:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8807.502611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8815.661518] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 8816.289253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8816.302980] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8816.513445] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8827.715950] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8833.606078] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8833.901244] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8837.884708] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8838.077801] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8842.221518] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8842.242338] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8842.277880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8842.292514] Lustre: Skipped 1 previous similar message [ 8846.422153] Lustre: server umount lustre-OST0000 complete [ 8852.455850] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8852.482483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8852.496380] Lustre: Skipped 2 previous similar messages [ 8857.578798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8857.597000] Lustre: Skipped 2 previous similar messages [ 8862.695035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8862.702858] Lustre: Skipped 1 previous similar message [ 8865.248185] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8865.538197] Lustre: server umount lustre-MDT0000 complete [ 8870.213692] LustreError: 285190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446053 with bad export cookie 5878078513180697974 [ 8870.214482] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8870.226715] LustreError: 285190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8870.873939] Lustre: server umount lustre-MDT0001 complete [ 8881.880547] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 10:34:23 (1788446063) [ 8891.682053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8892.254321] LustreError: 289583:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8892.286121] LustreError: 289583:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8892.384443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8896.954635] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 8897.505976] LustreError: 289584:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8902.641562] LustreError: 289583:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8907.366090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8907.738891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8912.168108] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 8920.369729] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8926.696827] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8933.706929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8934.046671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8940.922522] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 8949.145932] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8963.048186] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8963.049606] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8963.057774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8963.061307] LustreError: Skipped 1 previous similar message [ 8963.072269] Lustre: Skipped 1 previous similar message [ 8966.967879] Lustre: server umount lustre-OST0000 complete [ 8973.281636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8973.288189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8973.294123] Lustre: Skipped 1 previous similar message [ 8973.312561] Lustre: Skipped 2 previous similar messages [ 8977.815748] Lustre: server umount lustre-MDT0000 complete [ 8978.402441] LustreError: 291583:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8978.437753] LustreError: 291583:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8982.091084] LustreError: 289563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446165 with bad export cookie 5878078513180698856 [ 8982.101622] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8982.634074] Lustre: server umount lustre-MDT0001 complete [ 8994.350533] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 8997.292254] Key type lgssc unregistered [ 8997.790596] LNet: 293119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8997.808681] LNetError: 293119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8997.833819] LNet: Removed LNI 192.168.201.146@tcp [ 8999.164234] Key type .llcrypt unregistered [ 8999.165983] Key type ._llcrypt unregistered [ 9031.730613] Key type ._llcrypt registered [ 9031.739447] Key type .llcrypt registered [ 9031.841648] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 9047.482354] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 9048.759159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9048.970393] alg: No test for adler32 (adler32-zlib) [ 9050.038711] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9050.314633] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 9052.048279] Key type lgssc registered [ 9053.157211] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9095.917365] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9095.931138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9097.196360] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9097.227988] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9097.355626] Lustre: lustre-MDT0000: new disk, initializing [ 9097.486974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9097.525864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9101.763040] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 9112.744863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9112.975884] Lustre: 296945:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9113.152124] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9113.157846] Lustre: Skipped 1 previous similar message [ 9113.340055] Lustre: lustre-MDT0001: new disk, initializing [ 9113.544500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9113.646158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9113.664863] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9118.798536] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 9128.876356] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9134.533570] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9141.363659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9141.580563] Lustre: lustre-OST2710: new disk, initializing [ 9141.584558] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9141.588810] Lustre: 298915:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9141.648255] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9148.152309] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 9151.048441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9151.070284] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9151.230933] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9158.117928] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9165.416838] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9165.610853] Lustre: lustre-OST03e8: new disk, initializing [ 9165.621870] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9165.639884] Lustre: 299983:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9165.737291] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9172.553510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9172.576085] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9172.693775] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9172.915256] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 9181.925400] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9189.203933] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9189.423873] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9194.327768] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9194.669083] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9199.059539] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9199.262228] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9203.912153] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9204.094237] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9208.812539] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9208.815149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9208.821347] Lustre: Skipped 1 previous similar message [ 9213.629892] Lustre: server umount lustre-MDT0000 complete [ 9217.153842] LustreError: 296935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446400 with bad export cookie 2002378982771266462 [ 9217.166280] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9217.170765] LustreError: 296935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9217.463538] Lustre: server umount lustre-MDT0001 complete [ 9231.683986] Lustre: server umount lustre-OST2710 complete [ 9235.425268] Lustre: 294742:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788446402/real 1788446402] req@000000007bcc3aea x1875321797282304/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788446418 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9235.469667] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9235.484412] Lustre: Skipped 2 previous similar messages [ 9236.054903] Lustre: server umount lustre-OST03e8 complete [ 9256.313537] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 9265.595790] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 9300.445229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9300.659791] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9300.684963] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9300.745298] Lustre: lustre-MDT0000: new disk, initializing [ 9300.792880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9300.807991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9305.811839] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 9316.367567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9316.545387] Lustre: 305231:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9316.580366] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9316.583332] Lustre: Skipped 1 previous similar message [ 9316.654671] Lustre: lustre-MDT0001: new disk, initializing [ 9316.740845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9316.762191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9316.772749] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9322.251396] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 9332.298582] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9338.682925] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9345.953278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9346.236534] Lustre: lustre-OST0000: new disk, initializing [ 9346.239888] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9346.245335] Lustre: 307197:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9346.329657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9353.834458] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 9354.825850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9354.836022] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9354.971305] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9363.423113] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9368.741860] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9369.008193] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9373.547324] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9373.817259] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9380.837763] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9380.843307] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9380.853377] LustreError: Skipped 1 previous similar message [ 9380.862416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9380.873899] Lustre: Skipped 1 previous similar message [ 9380.894606] Lustre: Skipped 4 previous similar messages [ 9385.955287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9385.963238] Lustre: Skipped 1 previous similar message [ 9390.048134] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9390.336848] Lustre: server umount lustre-OST0000 complete [ 9396.200242] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9396.203666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9396.231039] Lustre: Skipped 1 previous similar message [ 9400.398350] Lustre: server umount lustre-MDT0000 complete [ 9401.320669] LustreError: 305242:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9401.352925] LustreError: 305242:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9405.252302] LustreError: 305221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446588 with bad export cookie 2002378982771268142 [ 9405.255628] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9405.266880] LustreError: 305221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9405.832473] Lustre: server umount lustre-MDT0001 complete [ 9415.175162] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 10:43:18 (1788446598) [ 9416.734658] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9418.614995] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 10:43:21 (1788446601) [ 9422.425806] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 9448.818234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9449.189374] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9449.219575] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9449.415922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9454.496685] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 9462.922460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9462.994546] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9463.015636] Lustre: 311059:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9467.957797] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 9490.930571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9491.119855] Lustre: Denying initial registration attempt from nid 192.168.201.146@tcp, specified as failover [ 9491.131136] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9491.148154] LustreError: 312513:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -99 [ 9491.156370] LustreError: 312513:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9491.161489] LustreError: 312513:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9491.268275] Lustre: server umount lustre-OST0000 complete [ 9491.274397] LustreError: 312513:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9499.108635] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9499.110312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9499.125877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9499.130307] Lustre: Skipped 2 previous similar messages [ 9504.500260] Lustre: server umount lustre-MDT0000 complete [ 9508.611980] LustreError: 311051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446691 with bad export cookie 2002378982771269024 [ 9508.615434] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9508.623074] LustreError: 311051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9509.160942] Lustre: server umount lustre-MDT0001 complete [ 9521.721353] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 9525.132640] Key type lgssc unregistered [ 9525.594414] LNet: 313635:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9525.599419] LNetError: 313635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9525.614905] LNet: Removed LNI 192.168.201.146@tcp [ 9526.986911] Key type .llcrypt unregistered [ 9526.989846] Key type ._llcrypt unregistered [ 9538.073381] Key type ._llcrypt registered [ 9538.076371] Key type .llcrypt registered [ 9538.149978] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 9539.040354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9539.052368] alg: No test for adler32 (adler32-zlib) [ 9540.005290] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9540.220135] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 9541.872569] Key type lgssc registered [ 9543.389069] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9589.111895] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 9597.497173] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9597.516643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9598.745730] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9598.761436] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9598.879779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9603.873665] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 9613.634959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9613.745866] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9613.767638] Lustre: 316395:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9614.073375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9619.133179] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 9626.372607] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9631.863175] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9651.943729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9652.210804] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9652.444842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9658.922177] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 9666.913403] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9669.602035] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9669.610539] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9669.621346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9670.113423] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9670.122428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9670.129232] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9670.161905] Lustre: Skipped 1 previous similar message [ 9675.240691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9675.255734] Lustre: Skipped 1 previous similar message [ 9675.535028] Lustre: server umount lustre-OST0000 complete [ 9680.359120] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9680.361383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9680.371794] Lustre: Skipped 1 previous similar message [ 9685.438135] Lustre: server umount lustre-MDT0000 complete [ 9688.962589] LustreError: 316388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446872 with bad export cookie 18134035310556305522 [ 9688.969393] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9688.981476] LustreError: 316388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9689.221377] Lustre: server umount lustre-MDT0001 complete [ 9700.333145] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 9702.821829] Key type lgssc unregistered [ 9703.138901] LNet: 319966:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9703.143455] LNetError: 319966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9704.171953] LNet: Removed LNI 192.168.201.146@tcp [ 9705.235273] Key type .llcrypt unregistered [ 9705.240120] Key type ._llcrypt unregistered [ 9729.692370] Key type ._llcrypt registered [ 9729.694391] Key type .llcrypt registered [ 9729.781862] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 9731.051826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9731.080917] alg: No test for adler32 (adler32-zlib) [ 9732.178885] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9732.468604] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 9734.128211] Key type lgssc registered [ 9735.276303] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9742.603834] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9742.624472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9744.213299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9749.226468] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 9754.594752] LustreError: 321380:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9754.608571] LustreError: 321380:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9757.862318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9758.416176] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9763.152731] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 9770.736563] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9776.639913] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9784.808163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9785.189635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9790.007420] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 9796.540339] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9813.987626] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9813.994837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9814.008902] Lustre: Skipped 2 previous similar messages [ 9814.021128] Lustre: Skipped 2 previous similar messages [ 9814.710473] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.46@tcp (stopping) [ 9817.221798] Lustre: server umount lustre-MDT0000 complete [ 9819.121134] LustreError: 323872:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9819.136270] LustreError: 323872:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9820.400583] LustreError: 321360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447003 with bad export cookie 12349936222599402141 [ 9820.416540] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9820.417502] LustreError: 321360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9824.225886] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9824.229752] LustreError: 321377:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9824.238692] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9824.247191] LustreError: 321377:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9826.708294] Lustre: server umount lustre-MDT0001 complete [ 9833.495874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9840.914118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9845.736497] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xab63c98fcac46fe6 [ 9845.743667] Lustre: MGC192.168.201.146@tcp: Connection restored to 0@lo (at 0@lo) [ 9845.982872] LustreError: 323212:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9845.997616] LustreError: 323212:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9846.031965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9846.085469] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9850.156699] LustreError: 324999:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9850.180339] LustreError: 324999:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9850.411683] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 9855.287713] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9857.976198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9858.337095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9862.987869] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 9863.682364] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9870.185905] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9871.729940] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9877.187582] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9879.042448] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9887.202191] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9887.214084] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9887.224458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9887.229149] Lustre: Skipped 2 previous similar messages [ 9889.257384] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9889.260308] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9890.663621] Lustre: server umount lustre-OST0000 complete [ 9894.373439] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9894.374463] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9894.379072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9894.379082] Lustre: Skipped 2 previous similar messages [ 9894.388458] Lustre: Skipped 1 previous similar message [ 9900.749562] Lustre: server umount lustre-MDT0000 complete [ 9904.612630] LustreError: 325002:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9904.649282] LustreError: 325002:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9905.441667] LustreError: 323225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447088 with bad export cookie 12349936222599409638 [ 9905.445965] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9905.448526] LustreError: 323225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9906.021413] Lustre: server umount lustre-MDT0001 complete [ 9917.125970] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 9921.731815] Key type lgssc unregistered [ 9922.141543] LNet: 327672:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9922.154602] LNetError: 327672:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9922.178489] LNet: Removed LNI 192.168.201.146@tcp [ 9923.295840] Key type .llcrypt unregistered [ 9923.305133] Key type ._llcrypt unregistered [ 9947.793231] Key type ._llcrypt registered [ 9947.796898] Key type .llcrypt registered [ 9947.947510] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 9949.087042] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9949.123149] alg: No test for adler32 (adler32-zlib) [ 9950.451028] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9950.717393] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 9952.577387] Key type lgssc registered [ 9953.457762] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9959.346138] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9959.355410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9960.632074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9965.390553] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 9970.156610] LustreError: 329086:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9970.188072] LustreError: 329086:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9974.769377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9975.273578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9981.581645] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 9989.604420] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9995.833184] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10000.034391] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10000.043687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10000.051835] Lustre: Skipped 1 previous similar message [10005.335664] Lustre: server umount lustre-MDT0000 complete [10005.474167] LustreError: 329086:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10005.480334] LustreError: 329086:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10008.464516] LustreError: 329067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447191 with bad export cookie 9600119015377390974 [10008.476553] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10008.748371] Lustre: server umount lustre-MDT0001 complete [10021.866143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10022.051228] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10022.075281] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10022.220165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10026.219350] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10034.216971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10034.289439] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10034.317262] Lustre: 331673:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10034.570344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10038.879927] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10046.404724] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10051.487397] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10060.843341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10061.099414] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10061.113214] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10061.121806] LustreError: 333556:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [10061.138615] LustreError: 333556:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [10061.145642] LustreError: 333556:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10061.203852] Lustre: server umount lustre-OST0000 complete [10061.206484] LustreError: 333556:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10071.415043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10071.693678] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10071.928960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10077.876585] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10079.558679] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10088.801371] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10095.913666] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10095.988901] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10096.037231] Lustre: lustre-OST0001: new disk, initializing [10096.039722] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10096.042691] Lustre: 335097:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10096.101684] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10101.918990] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10103.384352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10103.394764] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10103.458971] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10108.197847] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10114.031770] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10114.040374] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10114.065227] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10114.087777] Lustre: Skipped 2 previous similar messages [10119.139686] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10119.159468] Lustre: Skipped 2 previous similar messages [10124.270982] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10124.282692] Lustre: Skipped 1 previous similar message [10125.280389] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10125.548804] Lustre: server umount lustre-OST0001 complete [10129.379623] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10129.380657] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10129.390874] LustreError: Skipped 2 previous similar messages [10129.403636] LustreError: 334023:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10129.404074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10129.438374] LustreError: 334023:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10134.506089] LustreError: 335104:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10134.524912] LustreError: 335104:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10135.565604] Lustre: server umount lustre-OST0000 complete [10139.623893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10139.632854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10139.644638] Lustre: Skipped 2 previous similar messages [10139.661239] Lustre: Skipped 4 previous similar messages [10145.755383] Lustre: server umount lustre-MDT0000 complete [10148.955037] LustreError: 331665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447332 with bad export cookie 9600119015377391527 [10148.971214] LustreError: 331665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10148.975859] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10149.354672] Lustre: server umount lustre-MDT0001 complete [10158.362067] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [10161.502149] Key type lgssc unregistered [10161.929899] LNet: 337057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10161.938805] LNetError: 337057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10161.948837] LNet: Removed LNI 192.168.201.146@tcp [10162.910185] Key type .llcrypt unregistered [10162.916101] Key type ._llcrypt unregistered [10235.543136] Key type ._llcrypt registered [10235.544506] Key type .llcrypt registered [10235.663892] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [10248.444971] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [10249.311850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10249.487401] alg: No test for adler32 (adler32-zlib) [10250.628148] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [10250.831704] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [10252.480169] Key type lgssc registered [10253.700190] Lustre: Echo OBD driver; http://www.lustre.org/ [10292.433453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10292.457885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10293.755847] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10293.796959] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10293.910864] Lustre: lustre-MDT0000: new disk, initializing [10294.102896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10294.133970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10298.602798] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10309.981699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10310.102509] Lustre: 341971:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10310.162351] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10310.171943] Lustre: Skipped 1 previous similar message [10310.284377] Lustre: lustre-MDT0001: new disk, initializing [10310.406478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10310.447103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10310.465336] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10315.285716] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10323.984947] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10327.911705] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10334.370590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10334.867850] Lustre: lustre-OST0000: new disk, initializing [10334.877455] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10334.895984] Lustre: 343936:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10335.020209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10342.688328] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10343.049550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10343.062300] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10343.280469] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10352.482609] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10357.652862] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10357.835501] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10361.876434] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10362.027260] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10363.879059] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10363.880903] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10363.899424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10363.912391] Lustre: Skipped 1 previous similar message [10368.993415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10369.005667] Lustre: Skipped 2 previous similar messages [10370.016363] Lustre: server umount lustre-OST0000 complete [10377.185531] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10377.195104] LustreError: Skipped 1 previous similar message [10377.198965] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10377.214879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10379.238273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10379.245149] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10380.742168] Lustre: server umount lustre-MDT0000 complete [10385.027734] LustreError: 341964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447568 with bad export cookie 7513112876177347274 [10385.036744] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10385.044390] LustreError: 341964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10385.384725] Lustre: server umount lustre-MDT0001 complete [10393.136982] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 10:59:36 (1788447576) [10401.894495] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 10:59:44 (1788447584) [10411.193202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10411.641459] LustreError: 346692:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10411.661553] LustreError: 346692:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10411.741197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10416.284589] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10417.123542] LustreError: 346693:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10421.235429] LustreError: 346692:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10424.970113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10425.229236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10429.745082] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10437.109710] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10443.599761] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10451.148822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10451.452176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10457.604155] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10465.185943] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10477.521612] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10479.469584] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10481.183129] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10482.720850] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10484.496581] Lustre: Failing over lustre-MDT0000 [10484.916159] Lustre: server umount lustre-MDT0000 complete [10485.236704] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10485.257669] LustreError: 347707:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10485.259485] Lustre: Skipped 1 previous similar message [10485.293598] LustreError: 347707:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10490.337991] LustreError: 346692:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10490.357913] LustreError: 346692:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10498.715839] LustreError: 346689:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10498.734774] LustreError: 346689:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10501.601549] Lustre: 340191:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788447668/real 1788447668] req@00000000b810a8e0 x1875323056624256/t0(0) o400->MGC192.168.201.146@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788447684 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10501.626406] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10503.332567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10510.827916] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x6843f05acdf6da56 [10510.845983] Lustre: MGC192.168.201.146@tcp: Connection restored to 0@lo (at 0@lo) [10511.104485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10511.166209] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10513.067760] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10515.545859] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10516.469532] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10516.518229] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10516.553953] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10524.387846] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10525.971097] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10529.684146] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10531.512925] Lustre: Failing over lustre-MDT0000 [10531.809102] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10531.818498] LustreError: 349940:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10531.824285] Lustre: Skipped 3 previous similar messages [10531.835893] LustreError: 349940:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 16 previous similar messages [10533.845586] Lustre: server umount lustre-MDT0000 complete [10552.805724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10552.940850] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10553.242542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10553.305152] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10554.024676] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10557.843074] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10558.443894] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10558.448751] Lustre: Skipped 2 previous similar messages [10558.490889] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10558.557590] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10565.681505] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10566.987811] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10573.793443] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10573.796635] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10573.813591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10573.827713] Lustre: Skipped 1 previous similar message [10573.848192] Lustre: Skipped 3 previous similar messages [10577.927653] Lustre: server umount lustre-MDT0000 complete [10578.918360] LustreError: 346692:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10578.948981] LustreError: 346692:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [10581.114789] LustreError: 346674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447764 with bad export cookie 7513112876177349983 [10581.122928] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10581.126784] LustreError: 346674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10581.369458] Lustre: server umount lustre-MDT0001 complete [10595.511519] Lustre: server umount lustre-OST0000 complete [10610.582853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10616.558293] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10625.406106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10625.852804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10625.860216] Lustre: Skipped 1 previous similar message [10630.868810] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10640.285573] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10645.890591] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10652.810545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10656.528566] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10660.044430] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10667.878270] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10681.318913] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10681.338745] Lustre: Skipped 1 previous similar message [10681.342976] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10681.343437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10681.364679] Lustre: Skipped 1 previous similar message [10684.346417] Lustre: server umount lustre-OST0000 complete [10694.647855] Lustre: server umount lustre-MDT0000 complete [10696.673124] LustreError: 353794:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10696.687400] LustreError: 353794:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10698.027208] LustreError: 353047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447881 with bad export cookie 7513112876177350732 [10698.042346] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10698.045022] LustreError: 353047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10698.315536] Lustre: server umount lustre-MDT0001 complete [10709.125757] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [10711.806852] Key type lgssc unregistered [10712.087908] LNet: 356598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10712.092867] LNetError: 356598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10712.106537] LNet: Removed LNI 192.168.201.146@tcp [10712.919106] Key type .llcrypt unregistered [10712.922737] Key type ._llcrypt unregistered [10738.941141] Key type ._llcrypt registered [10738.943205] Key type .llcrypt registered [10739.100622] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [10740.287146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10740.329103] alg: No test for adler32 (adler32-zlib) [10741.518328] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [10741.850260] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [10743.560224] Key type lgssc registered [10744.993512] Lustre: Echo OBD driver; http://www.lustre.org/ [10754.124797] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10754.143168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10755.845535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10762.815928] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10766.307413] LustreError: 358106:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10766.330571] LustreError: 358106:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10771.427978] LustreError: 358105:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10771.548991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10771.960288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10776.676569] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10784.086681] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10788.897902] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10794.991469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10795.405467] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10801.205577] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10801.592751] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10807.263305] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10818.019692] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10818.026522] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10818.042529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10822.117107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10822.117914] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10823.994133] Lustre: server umount lustre-MDT0000 complete [10827.246463] LustreError: 359110:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10827.263117] LustreError: 359110:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10828.347493] LustreError: 358085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448011 with bad export cookie 5595317208737404190 [10828.355656] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10828.361562] LustreError: 358085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10828.518880] Lustre: server umount lustre-MDT0001 complete [10842.205985] Lustre: server umount lustre-OST0000 complete [10853.991848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10854.234183] LustreError: 361816:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10854.299888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10858.765704] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10859.504958] LustreError: 361817:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10867.124517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10867.355724] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10872.658540] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10880.718612] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10886.502574] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10892.139857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10892.511145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10896.405847] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10899.715153] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10904.475736] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10917.358753] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10917.365171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10917.377936] Lustre: Skipped 3 previous similar messages [10917.397548] Lustre: Skipped 4 previous similar messages [10922.430664] Lustre: server umount lustre-MDT0000 complete [10922.473425] LustreError: 361812:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10922.498332] LustreError: 361812:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10926.337304] LustreError: 361796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448109 with bad export cookie 5595317208737405303 [10926.341409] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10926.345566] LustreError: 361796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10926.667684] Lustre: server umount lustre-MDT0001 complete [10940.038656] Lustre: server umount lustre-OST0000 complete [10954.079755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10954.398358] LustreError: 365727:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10954.494469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10958.741794] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10966.171839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10971.816110] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10979.576703] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10986.159542] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10992.494925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10992.725241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10992.728945] Lustre: Skipped 1 previous similar message [10997.284466] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10998.897982] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11002.923277] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11009.509757] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11009.514730] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11009.573748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11013.600413] Lustre: server umount lustre-OST0000 complete [11017.697720] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11017.703128] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11017.722182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11017.722191] Lustre: Skipped 2 previous similar messages [11023.134561] Lustre: server umount lustre-MDT0000 complete [11026.523398] LustreError: 368505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448209 with bad export cookie 5595317208737406318 [11026.529039] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11026.533947] LustreError: 368505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11026.792103] Lustre: server umount lustre-MDT0001 complete [11037.980912] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [11040.879255] Key type lgssc unregistered [11041.185865] LNet: 369258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11041.200090] LNetError: 369258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11041.220827] LNet: Removed LNI 192.168.201.146@tcp [11042.078833] Key type .llcrypt unregistered [11042.079953] Key type ._llcrypt unregistered [11067.287890] Key type ._llcrypt registered [11067.291888] Key type .llcrypt registered [11067.398229] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [11068.684932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11068.766655] alg: No test for adler32 (adler32-zlib) [11069.937259] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11070.225475] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [11071.936241] Key type lgssc registered [11073.563415] Lustre: Echo OBD driver; http://www.lustre.org/ [11082.709273] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11082.726941] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11082.738779] LustreError: 370696:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11082.756035] LustreError: 370696:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11082.764584] LustreError: 370696:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11082.774225] LustreError: 370696:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11082.790519] LustreError: 370696:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11090.691968] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11090.696562] LustreError: 371094:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11090.712965] LustreError: 371094:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11090.720968] LustreError: 371094:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11090.729896] LustreError: 371094:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11090.739822] LustreError: 371094:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11112.023451] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [11114.804394] Key type lgssc unregistered [11115.038423] LNet: 372095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11115.043509] LNetError: 372095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11115.062881] LNet: Removed LNI 192.168.201.146@tcp [11115.732541] Key type .llcrypt unregistered [11115.738750] Key type ._llcrypt unregistered [11135.630120] Key type ._llcrypt registered [11135.631576] Key type .llcrypt registered [11135.738435] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [11152.451902] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [11153.510265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11153.656879] alg: No test for adler32 (adler32-zlib) [11154.869480] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11155.177379] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [11156.904168] Key type lgssc registered [11158.059198] Lustre: Echo OBD driver; http://www.lustre.org/ [11193.737718] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11193.799061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11195.220517] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11195.277526] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11195.421409] Lustre: lustre-MDT0000: new disk, initializing [11195.581751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11195.623843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11202.595804] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [11213.450757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11213.682826] Lustre: 375246:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11213.793092] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11213.804842] Lustre: Skipped 1 previous similar message [11213.895819] Lustre: lustre-MDT0001: new disk, initializing [11213.956818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11213.991973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11214.020662] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11219.682233] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [11232.233148] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11238.368299] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11245.398533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11245.670468] Lustre: lustre-OST0000: new disk, initializing [11245.676431] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11245.701656] Lustre: 377216:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11245.851726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11252.193342] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [11253.288899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11253.296838] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11253.448357] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11261.554456] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11266.420270] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11266.790498] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11272.024696] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11272.388574] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11279.332691] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11279.337961] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11279.359332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11279.370965] Lustre: Skipped 1 previous similar message [11280.743375] Lustre: server umount lustre-OST0000 complete [11285.989952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11285.995991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11286.006198] Lustre: Skipped 1 previous similar message [11286.019503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11289.570986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11289.571596] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11291.169812] Lustre: server umount lustre-MDT0000 complete [11295.307970] LustreError: 375239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448478 with bad export cookie 12886667882172468634 [11295.320679] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11295.329075] LustreError: 375239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11295.731928] Lustre: server umount lustre-MDT0001 complete [11304.773698] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11306.509734] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 11:14:49 (1788448489) [11313.095875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11313.472675] LustreError: 379676:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11313.498469] LustreError: 379676:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11313.551398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11318.286598] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [11318.754231] LustreError: 379677:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11322.855829] LustreError: 379676:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11327.411749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11327.721673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11332.581135] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [11339.721980] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11345.435266] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11351.855377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11352.137051] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11358.336654] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [11365.213858] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11372.273262] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11372.471338] Lustre: lustre-OST0001: new disk, initializing [11372.475903] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11372.491099] Lustre: 382511:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11372.614242] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11378.110373] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [11380.292149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11380.298154] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11380.363624] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11387.626787] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11395.774192] Lustre: lustre-OST0001: Not available for connect from 192.168.201.46@tcp (stopping) [11395.783214] Lustre: Skipped 1 previous similar message [11396.067332] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11396.084586] Lustre: Skipped 1 previous similar message [11398.546580] Lustre: server umount lustre-OST0001 complete [11400.858674] LustreError: 381496:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11406.310188] LustreError: 381497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11406.337995] LustreError: 381497:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11416.560858] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11416.567154] LustreError: 382116:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11416.580834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11416.588535] LustreError: 382116:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11416.591631] Lustre: Skipped 2 previous similar messages [11421.610566] Lustre: server umount lustre-OST0000 complete [11425.250525] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11425.258851] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11425.284101] Lustre: Skipped 1 previous similar message [11425.294412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11425.300307] Lustre: Skipped 1 previous similar message [11431.322257] Lustre: server umount lustre-MDT0000 complete [11434.668958] LustreError: 382510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448617 with bad export cookie 12886667882172469516 [11434.674181] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11434.679470] LustreError: 382510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11440.999378] Lustre: server umount lustre-MDT0001 complete [11451.302489] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [11454.172584] Key type lgssc unregistered [11454.473890] LNet: 384506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11454.479077] LNetError: 384506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11454.491978] LNet: Removed LNI 192.168.201.146@tcp [11455.614640] Key type .llcrypt unregistered [11455.618221] Key type ._llcrypt unregistered [11509.577567] Key type ._llcrypt registered [11509.580854] Key type .llcrypt registered [11509.817233] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [11511.900642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11512.039127] alg: No test for adler32 (adler32-zlib) [11513.188276] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11513.445351] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [11515.088220] Key type lgssc registered [11516.318651] Lustre: Echo OBD driver; http://www.lustre.org/ [11523.605375] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11523.629439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11524.914466] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11524.928658] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11525.042165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11529.509372] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [11538.643190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11538.740252] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11538.785137] Lustre: 387302:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11539.148392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11544.047510] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [11552.057941] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11557.349700] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11559.910704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11559.947729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11559.961731] Lustre: Skipped 1 previous similar message [11565.027297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11565.031562] Lustre: Skipped 1 previous similar message [11565.717606] Lustre: server umount lustre-MDT0000 complete [11569.541322] LustreError: 387295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448752 with bad export cookie 15718187681756411196 [11569.549365] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11569.843279] Lustre: server umount lustre-MDT0001 complete [11578.483885] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 11:19:21 (1788448761) [11584.888072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11585.368588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11590.425668] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [11590.640851] LustreError: 389815:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11590.663221] LustreError: 389815:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11595.752521] LustreError: 389816:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11598.805633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11599.210863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11603.612535] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [11611.150827] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11616.688170] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11624.445790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11624.656359] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11624.838173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11631.563875] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [11639.496886] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11647.644912] Lustre: Permanently deactivating lustre-OST0000 [11647.681772] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11654.183127] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11655.522313] LustreError: 392756:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11655.529829] LustreError: 392756:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11655.537378] LustreError: 392756:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11658.391557] LustreError: 392805:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11658.396934] LustreError: 392805:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11666.520970] Lustre: server umount lustre-OST0000 complete [11669.353665] LustreError: 393054:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11669.360886] LustreError: 393054:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11671.523663] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11671.541385] Lustre: Skipped 1 previous similar message [11671.549021] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11676.129834] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11676.144054] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11676.157810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11680.814850] Lustre: server umount lustre-MDT0000 complete [11681.761298] LustreError: 390540:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11681.784906] LustreError: 390540:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11684.484695] LustreError: 391649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448867 with bad export cookie 15718187681756411777 [11684.507847] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11684.510258] LustreError: 391649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [11694.244141] Lustre: server umount lustre-MDT0001 complete [11702.990822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11707.546740] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [11710.623530] LustreError: 394214:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11717.393690] Lustre: 394554:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.146@tcp [11719.820405] Lustre: server umount MGS complete [11731.729023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11732.254075] LustreError: 395279:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11732.266717] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11732.270613] Lustre: Skipped 1 previous similar message [11732.343441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11737.432478] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [11737.580923] LustreError: 395280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11747.320206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11747.679553] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11752.832466] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [11760.646221] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11766.381649] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11768.011643] Lustre: Permanently reactivating lustre-OST0000 [11768.026478] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11768.031346] Lustre: Skipped 2 previous similar messages [11794.913161] LustreError: 397046:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11802.609611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11803.069199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11803.076931] Lustre: Skipped 1 previous similar message [11810.019608] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [11817.929160] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11824.610331] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11824.622951] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11824.640170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11824.642736] Lustre: Skipped 1 previous similar message [11825.350954] Lustre: server umount lustre-OST0000 complete [11829.729849] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11829.739714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11835.884759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11835.889308] Lustre: Skipped 4 previous similar messages [11838.996623] Lustre: server umount lustre-MDT0000 complete [11839.970278] LustreError: 395276:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11840.021728] LustreError: 395276:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11843.558857] LustreError: 395260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449026 with bad export cookie 15718187681756412827 [11843.562646] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11843.579066] LustreError: 395260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11844.036802] Lustre: server umount lustre-MDT0001 complete [11854.739389] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [11858.679813] Key type lgssc unregistered [11858.981787] LNet: 399641:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11859.002408] LNetError: 399641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11859.022261] LNet: Removed LNI 192.168.201.146@tcp [11859.881693] Key type .llcrypt unregistered [11859.883931] Key type ._llcrypt unregistered [11879.497448] Key type ._llcrypt registered [11879.503971] Key type .llcrypt registered [11879.651963] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [11894.363935] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [11895.752822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11895.836702] alg: No test for adler32 (adler32-zlib) [11896.867348] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11897.152258] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [11898.848210] Key type lgssc registered [11899.914476] Lustre: Echo OBD driver; http://www.lustre.org/ [11936.722169] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11936.766305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11937.974707] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11938.027043] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11938.139434] Lustre: lustre-MDT0000: new disk, initializing [11938.340863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11938.391207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11943.382667] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [11953.631042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11953.751221] Lustre: 402799:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11953.796712] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11953.800524] Lustre: Skipped 1 previous similar message [11953.862624] Lustre: lustre-MDT0001: new disk, initializing [11953.962772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11953.988875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11953.999633] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11957.330888] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [11966.076511] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11972.119380] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11979.088563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11979.514868] Lustre: lustre-OST0000: new disk, initializing [11979.517674] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11979.526885] Lustre: 404765:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11979.623805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11987.021424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11987.052948] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11987.100558] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [11987.214900] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11995.828442] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12000.326940] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12000.528199] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12005.022953] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12005.222457] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12007.908634] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12007.910239] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12007.918425] Lustre: Skipped 1 previous similar message [12007.924332] LustreError: Skipped 1 previous similar message [12007.924971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12007.950683] Lustre: Skipped 1 previous similar message [12013.036277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12013.045915] Lustre: Skipped 1 previous similar message [12013.460364] Lustre: server umount lustre-OST0000 complete [12018.146850] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12018.156710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12018.167982] Lustre: Skipped 1 previous similar message [12018.184533] Lustre: Skipped 1 previous similar message [12023.275666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12023.767764] Lustre: server umount lustre-MDT0000 complete [12027.362972] LustreError: 402790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449210 with bad export cookie 5909186564518470337 [12027.365602] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12027.374158] LustreError: 402790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12027.743455] Lustre: server umount lustre-MDT0001 complete [12035.718673] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12037.457242] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 11:27:00 (1788449220) [12046.011400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12046.735199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12051.939946] LustreError: 407225:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12051.995493] LustreError: 407225:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12052.520127] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [12056.034485] LustreError: 407226:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12061.155943] LustreError: 407225:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12061.421453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12061.884141] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12065.995673] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [12073.487823] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12080.172624] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12086.916847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12087.333626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12092.574599] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [12099.654596] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12102.087538] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12104.787524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12111.853686] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12111.858594] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12111.868600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12111.886536] Lustre: Skipped 1 previous similar message [12111.896315] Lustre: Skipped 1 previous similar message [12115.697327] Lustre: server umount lustre-OST0000 complete [12122.082062] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12122.098940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12122.107400] Lustre: Skipped 1 previous similar message [12125.579700] Lustre: server umount lustre-MDT0000 complete [12127.205981] LustreError: 407220:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12127.228678] LustreError: 407220:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12129.047553] LustreError: 407207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449312 with bad export cookie 5909186564518471219 [12129.055741] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12129.061559] LustreError: 407207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12129.280293] Lustre: server umount lustre-MDT0001 complete [12138.816804] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [12141.205855] Key type lgssc unregistered [12141.479883] LNet: 410948:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12141.484093] LNetError: 410948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12142.504942] LNet: Removed LNI 192.168.201.146@tcp [12143.181367] Key type .llcrypt unregistered [12143.183245] Key type ._llcrypt unregistered [12164.456370] Key type ._llcrypt registered [12164.458218] Key type .llcrypt registered [12164.543577] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [12164.741694] Key type .llcrypt unregistered [12164.743568] Key type ._llcrypt unregistered [12183.078885] Key type ._llcrypt registered [12183.081926] Key type .llcrypt registered [12183.274351] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [12184.511985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12184.542829] alg: No test for adler32 (adler32-zlib) [12185.720857] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12185.996276] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [12187.752804] Key type lgssc registered [12189.176185] Lustre: Echo OBD driver; http://www.lustre.org/ [12196.722647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12196.737903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12198.216553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12203.125562] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [12208.615451] LustreError: 413155:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12208.632400] LustreError: 413155:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12212.294660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12212.609262] LustreError: 413156:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12212.646121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12217.828107] LustreError: 413155:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12217.858545] LustreError: 413155:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12220.298608] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [12222.953789] LustreError: 413156:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12222.973854] LustreError: 413156:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12222.991652] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12228.070884] LustreError: 413150:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12228.081882] LustreError: 413150:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12228.255507] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12235.683570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12236.074199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12241.393381] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [12251.632616] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12251.641066] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12251.653879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12251.660205] Lustre: Skipped 1 previous similar message [12251.672962] Lustre: Skipped 1 previous similar message [12256.104497] Lustre: server umount lustre-OST0000 complete [12261.858356] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12261.870490] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12261.884835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12261.898300] Lustre: Skipped 1 previous similar message [12266.253492] Lustre: server umount lustre-MDT0000 complete [12266.988485] LustreError: 414891:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12267.021728] LustreError: 414891:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12269.243931] LustreError: 413137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449452 with bad export cookie 1810522047037517595 [12269.247528] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12269.260217] LustreError: 413137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12269.578897] Lustre: server umount lustre-MDT0001 complete [12279.544957] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [12281.934944] Key type lgssc unregistered [12282.300433] LNet: 416303:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12282.305538] LNetError: 416303:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12282.321556] LNet: Removed LNI 192.168.201.146@tcp [12283.255656] Key type .llcrypt unregistered [12283.257759] Key type ._llcrypt unregistered [12304.854181] Key type ._llcrypt registered [12304.856609] Key type .llcrypt registered [12304.960861] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [12305.740421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12305.810576] alg: No test for adler32 (adler32-zlib) [12307.074254] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12307.283536] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [12308.936258] Key type lgssc registered [12310.070422] Lustre: Echo OBD driver; http://www.lustre.org/ [12318.000545] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12318.024421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12333.600513] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12338.720325] LustreError: 417689:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.146@tcp: failed processing log, type 4: rc = -110 [12365.410616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12372.219814] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [12379.687779] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12386.450037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12387.010433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12387.102467] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12392.394467] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [12392.421593] LustreError: 417714:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12392.437644] LustreError: 417714:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12397.547627] LustreError: 417715:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12400.629773] LustreError: 417714:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12400.654956] LustreError: 417714:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12403.638439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12404.138762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12410.059848] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [12417.730904] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12423.280912] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12432.186768] Lustre: server umount lustre-OST0000 complete [12437.989402] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12437.996966] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12438.008236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12440.033601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12440.048815] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12445.155972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12445.161300] Lustre: Skipped 2 previous similar messages [12446.276814] Lustre: server umount lustre-MDT0000 complete [12449.474269] LustreError: 417696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449632 with bad export cookie 6642412207497530085 [12449.478858] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12449.479711] LustreError: 417696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12459.130318] Lustre: server umount lustre-MDT0001 complete [12469.174716] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [12472.056672] Key type lgssc unregistered [12472.289596] LNet: 421259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12472.300304] LNetError: 421259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12472.326502] LNet: Removed LNI 192.168.201.146@tcp [12473.070341] Key type .llcrypt unregistered [12473.077802] Key type ._llcrypt unregistered [12498.939657] Key type ._llcrypt registered [12498.941056] Key type .llcrypt registered [12499.029271] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [12500.388226] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12500.422357] alg: No test for adler32 (adler32-zlib) [12501.478645] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12501.802188] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [12503.505454] Key type lgssc registered [12504.942743] Lustre: Echo OBD driver; http://www.lustre.org/ [12513.898795] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12513.914433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12515.483177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12519.620120] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [12525.034510] LustreError: 422674:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12525.078869] LustreError: 422674:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12529.419803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12530.295792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12537.206719] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [12545.341022] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12551.437729] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12559.404056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12559.952053] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12561.028116] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12568.217400] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [12574.617778] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12580.839453] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12580.843413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12580.869508] Lustre: Skipped 2 previous similar messages [12580.883324] Lustre: Skipped 2 previous similar messages [12582.580084] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.46@tcp (stopping) [12585.905651] Lustre: server umount lustre-MDT0000 complete [12585.955163] LustreError: 422674:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12585.972126] LustreError: 422674:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12587.677131] LustreError: 422670:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12588.230258] LustreError: 422654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.46@tcp arrived at 1788449771 with bad export cookie 6548435491684590028 [12588.241959] LustreError: 422654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12589.462364] LustreError: 422655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449772 with bad export cookie 6548435491684589132 [12589.469579] LustreError: 422655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12589.474941] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12591.073823] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12591.074271] LustreError: 424791:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12591.094633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12591.103032] LustreError: 424791:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12594.082722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12595.964660] Lustre: server umount lustre-OST0000 complete [12599.208126] LustreError: 422673:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12599.246380] LustreError: 422673:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12603.424290] LustreError: 423403:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000030bb7aef x1875325416931584/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12603.652547] Lustre: server umount lustre-MDT0001 complete [12614.175305] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [12618.226344] Key type lgssc unregistered [12618.998599] LNet: 426357:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12619.012957] LNetError: 426357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12620.078351] LNet: Removed LNI 192.168.201.146@tcp [12621.677916] Key type .llcrypt unregistered [12621.684588] Key type ._llcrypt unregistered [12650.378443] Key type ._llcrypt registered [12650.380066] Key type .llcrypt registered [12650.463591] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [12651.724692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12651.849664] alg: No test for adler32 (adler32-zlib) [12653.068776] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12653.496458] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [12655.248182] Key type lgssc registered [12656.380672] Lustre: Echo OBD driver; http://www.lustre.org/ [12664.210825] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12664.220370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12665.594144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12670.182337] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [12675.054082] LustreError: 427771:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12675.082591] LustreError: 427771:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12678.498146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12678.813032] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12683.265468] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [12690.658163] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12695.487249] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12701.826302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12702.072039] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12709.819642] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [12710.278787] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12719.257359] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12725.222422] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12725.234502] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12725.253265] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12725.752693] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12725.777882] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12726.971384] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.46@tcp (stopping) [12730.337848] Lustre: server umount lustre-MDT0001 complete [12730.860635] LustreError: 427771:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12730.899033] LustreError: 427771:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12732.066263] LustreError: 427766:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12735.982366] LustreError: 427771:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12736.002320] LustreError: 427771:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12741.092407] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12741.098205] LustreError: 427772:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12741.110904] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12741.149603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12743.311917] Lustre: server umount lustre-OST0000 complete [12747.999992] Lustre: server umount lustre-MDT0000 complete [12761.381628] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [12764.468913] Key type lgssc unregistered [12764.784335] LNet: 431454:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12764.790752] LNetError: 431454:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12765.863795] LNet: Removed LNI 192.168.201.146@tcp [12767.050826] Key type .llcrypt unregistered [12767.053741] Key type ._llcrypt unregistered [12795.055368] Key type ._llcrypt registered [12795.058381] Key type .llcrypt registered [12795.199709] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [12811.320500] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [12813.002091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12813.045861] alg: No test for adler32 (adler32-zlib) [12814.200404] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12814.469645] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [12816.184660] Key type lgssc registered [12817.903700] Lustre: Echo OBD driver; http://www.lustre.org/ [12860.578178] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [12878.020959] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12878.041973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12879.462547] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12879.514766] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12879.656496] Lustre: lustre-MDT0000: new disk, initializing [12879.757188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12879.788457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12885.073193] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [12902.621043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12902.956662] Lustre: lustre-OST0000: new disk, initializing [12902.961689] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12902.965686] Lustre: Skipped 1 previous similar message [12902.974239] Lustre: 436492:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12903.094027] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12910.144225] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [12911.242952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12911.261839] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12911.297450] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12920.876720] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12936.967807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12937.140798] Lustre: 435471:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12937.277739] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12937.347456] Lustre: lustre-MDT0001: new disk, initializing [12937.480496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12937.536727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12937.549695] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12941.791416] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [12951.149311] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12951.406031] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12955.888508] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12956.082831] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12960.737975] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12960.746096] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12960.790708] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12960.796766] Lustre: Skipped 1 previous similar message [12963.007706] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.46@tcp (stopping) [12965.866709] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12965.875414] Lustre: Skipped 1 previous similar message [12967.045735] Lustre: server umount lustre-MDT0001 complete [12976.101192] LustreError: 439061:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12976.127840] LustreError: 439061:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12981.222217] LustreError: 435484:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12981.235896] LustreError: 435484:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12986.356936] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12986.366775] LustreError: 435483:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12986.370949] Lustre: Skipped 1 previous similar message [12986.373588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12991.461894] LustreError: 436474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12991.624801] Lustre: server umount lustre-OST0000 complete [13001.993170] Lustre: server umount lustre-MDT0000 complete [13014.760196] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [13017.446432] Key type lgssc unregistered [13017.645679] LNet: 440234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13017.651179] LNetError: 440234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13018.736151] LNet: Removed LNI 192.168.201.146@tcp [13019.467549] Key type .llcrypt unregistered [13019.471061] Key type ._llcrypt unregistered [13057.353343] Key type ._llcrypt registered [13057.355212] Key type .llcrypt registered [13057.459676] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [13057.700774] Key type .llcrypt unregistered [13057.702122] Key type ._llcrypt unregistered [13074.301853] Key type ._llcrypt registered [13074.303033] Key type .llcrypt registered [13074.456687] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [13075.465892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13075.522370] alg: No test for adler32 (adler32-zlib) [13076.686700] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13076.964413] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [13078.673151] Key type lgssc registered [13079.982245] Lustre: Echo OBD driver; http://www.lustre.org/ [13108.924325] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13108.957680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13110.276326] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13110.342717] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13110.517588] Lustre: lustre-MDT0000: new disk, initializing [13110.612902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13110.641373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13115.690755] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [13125.719308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13125.851771] Lustre: 443606:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13125.905985] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13125.911756] Lustre: Skipped 1 previous similar message [13125.990383] Lustre: lustre-MDT0001: new disk, initializing [13126.063963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13126.096247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13126.110432] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13130.152933] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [13139.597379] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13145.844910] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13151.872806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13152.049917] Lustre: lustre-OST0000: new disk, initializing [13152.053056] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13152.057244] Lustre: 445576:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13152.125319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13159.705755] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [13160.081887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13160.099181] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13160.220658] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13170.744091] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13177.315577] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13177.328256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13177.365321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13180.900338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13180.902346] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13182.831801] Lustre: server umount lustre-MDT0000 complete [13186.449024] LustreError: 443596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450369 with bad export cookie 18359464858147171519 [13186.457283] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13186.468753] LustreError: 443596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13186.872453] Lustre: server umount lustre-MDT0001 complete [13200.923865] Lustre: server umount lustre-OST0000 complete [13211.961820] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 11:46:34 (1788450394) [13219.454520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13220.129865] LustreError: 447606:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13220.156148] LustreError: 447606:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13220.230300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13225.031123] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [13225.445083] LustreError: 447607:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13230.564804] LustreError: 447606:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13233.473226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13234.114425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13239.749270] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [13247.780255] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13253.453701] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13273.079197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13273.356660] Lustre: Found index 0 for lustre-OST0000, updating log [13273.525559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13278.893236] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [13285.481874] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13290.936824] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13296.102358] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13296.112660] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13296.133495] Lustre: Skipped 2 previous similar messages [13296.135977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13296.150855] Lustre: Skipped 3 previous similar messages [13297.676500] Lustre: server umount lustre-OST0000 complete [13313.506434] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13313.507372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13313.523306] Lustre: Skipped 1 previous similar message [13313.526284] Lustre: Skipped 2 previous similar messages [13318.625560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13318.639425] Lustre: Skipped 1 previous similar message [13319.712527] Lustre: server umount lustre-MDT0000 complete [13323.309313] LustreError: 447586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450506 with bad export cookie 18359464858147172912 [13323.326820] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13323.330328] LustreError: 447586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13323.749850] Lustre: server umount lustre-MDT0001 complete [13331.248211] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 11:48:34 (1788450514) [13340.130568] LNet: 1 local NIs in recovery (showing 1): 192.168.201.146@tcp [13951.346671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13951.859661] LustreError: 454017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13951.945418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13956.977423] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [13957.100625] LustreError: 454018:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13962.212500] LustreError: 454017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13965.593693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13966.082666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13970.372924] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [13979.512769] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13986.494270] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14005.728601] LNet: 1 local NIs in recovery (showing 1): 192.168.201.146@tcp [14006.456754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14006.770745] Lustre: Found index 0 for lustre-OST0000, updating log [14007.114567] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14012.609576] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [14020.545096] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14022.516180] Lustre: server umount lustre-OST0000 complete [14026.416879] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14026.424205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14027.489976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14027.493308] Lustre: Skipped 2 previous similar messages [14029.537955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14029.547929] Lustre: Skipped 3 previous similar messages [14032.723370] Lustre: server umount lustre-MDT0000 complete [14033.636740] LustreError: 454017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14033.646070] LustreError: 454017:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14036.031610] LustreError: 455868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451219 with bad export cookie 18359464858147173997 [14036.034943] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14036.036246] LustreError: 455868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14036.364172] Lustre: server umount lustre-MDT0001 complete [14065.910123] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [14066.146331] LNet: 1 local NIs in recovery (showing 1): 192.168.201.146@tcp [14078.740575] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [14121.087482] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 12:01:43 (1788451303) [14139.337591] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [14142.513327] Key type lgssc unregistered [14142.776947] LNet: 460640:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14142.787552] LNetError: 460640:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14267.756723] LNet: Removed LNI 192.168.201.146@tcp [14268.637424] Key type .llcrypt unregistered [14268.639421] Key type ._llcrypt unregistered [14285.249296] Key type ._llcrypt registered [14285.251495] Key type .llcrypt registered [14285.350059] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [14286.940031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14287.011022] alg: No test for adler32 (adler32-zlib) [14288.200435] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [14288.482861] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [14290.161256] Key type lgssc registered [14291.269622] Lustre: Echo OBD driver; http://www.lustre.org/ [14295.501468] LNet: Added LNI 192.168.201.146@tcp100 [8/256/0/180] [14296.326305] LNet: Added LNI 192.168.201.146@tcp101 [8/256/0/180] [14297.258739] LNet: Added LNI 192.168.201.146@tcp102 [8/256/0/180] [14298.045476] LNet: Added LNI 192.168.201.146@tcp103 [8/256/0/180] [14298.936233] LNet: Added LNI 192.168.201.146@tcp104 [8/256/0/180] [14299.905633] LNet: Added LNI 192.168.201.146@tcp105 [8/256/0/180] [14300.810935] LNet: Added LNI 192.168.201.146@tcp106 [8/256/0/180] [14301.885460] LNet: Added LNI 192.168.201.146@tcp107 [8/256/0/180] [14302.860945] LNet: Added LNI 192.168.201.146@tcp108 [8/256/0/180] [14303.879268] LNet: Added LNI 192.168.201.146@tcp109 [8/256/0/180] [14305.027767] LNet: Added LNI 192.168.201.146@tcp110 [8/256/0/180] [14305.815780] LNet: Added LNI 192.168.201.146@tcp111 [8/256/0/180] [14306.679943] LNet: Added LNI 192.168.201.146@tcp112 [8/256/0/180] [14307.485670] LNet: Added LNI 192.168.201.146@tcp113 [8/256/0/180] [14308.296431] LNet: Added LNI 192.168.201.146@tcp114 [8/256/0/180] [14309.255938] LNet: Added LNI 192.168.201.146@tcp115 [8/256/0/180] [14310.310345] LNet: Added LNI 192.168.201.146@tcp116 [8/256/0/180] [14311.242691] LNet: Added LNI 192.168.201.146@tcp117 [8/256/0/180] [14312.242589] LNet: Added LNI 192.168.201.146@tcp118 [8/256/0/180] [14313.221323] LNet: Added LNI 192.168.201.146@tcp119 [8/256/0/180] [14314.462038] LNet: Added LNI 192.168.201.146@tcp120 [8/256/0/180] [14315.480194] LNet: Added LNI 192.168.201.146@tcp121 [8/256/0/180] [14316.585404] LNet: Added LNI 192.168.201.146@tcp122 [8/256/0/180] [14317.449503] LNet: Added LNI 192.168.201.146@tcp123 [8/256/0/180] [14318.462842] LNet: Added LNI 192.168.201.146@tcp124 [8/256/0/180] [14319.559386] LNet: Added LNI 192.168.201.146@tcp125 [8/256/0/180] [14320.521524] LNet: Added LNI 192.168.201.146@tcp126 [8/256/0/180] [14321.595696] LNet: Added LNI 192.168.201.146@tcp127 [8/256/0/180] [14322.554067] LNet: Added LNI 192.168.201.146@tcp128 [8/256/0/180] [14323.564440] LNet: Added LNI 192.168.201.146@tcp129 [8/256/0/180] [14324.638577] LNet: Added LNI 192.168.201.146@tcp130 [8/256/0/180] [14325.679506] LNet: Added LNI 192.168.201.146@tcp131 [8/256/0/180] [14326.686355] LNet: Added LNI 192.168.201.146@tcp132 [8/256/0/180] [14327.577655] LNet: Added LNI 192.168.201.146@tcp133 [8/256/0/180] [14328.442052] LNet: Added LNI 192.168.201.146@tcp134 [8/256/0/180] [14329.425107] LNet: Added LNI 192.168.201.146@tcp135 [8/256/0/180] [14336.691624] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14336.723124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14337.889193] Lustre: 463671:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.146@tcp: can't add NID 192.168.201.146@tcp130: rc = -75 [14337.951064] LNetError: 461129:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.146 [14337.964181] LNetError: Protocol error connecting to 192.168.201.146@tcp116 at host 192.168.201.146:988: is it running a compatible version of Lustre? [14338.104417] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14338.127312] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14338.213969] Lustre: lustre-MDT0000: new disk, initializing [14338.287633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14338.307676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14338.656352] LNet: 8 local NIs in recovery (showing 8): 192.168.201.146@tcp116, 192.168.201.146@tcp122, 192.168.201.146@tcp125, 192.168.201.146@tcp126, 192.168.201.146@tcp128, 192.168.201.146@tcp129, 192.168.201.146@tcp124, 192.168.201.146@tcp127 [14338.660615] LNetError: 463693:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.146 [14338.696961] LNetError: 463693:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 13 previous similar messages [14338.699353] LNetError: Protocol error connecting to 192.168.201.146@tcp103 at host 192.168.201.146:988: is it running a compatible version of Lustre? [14338.724366] LNetError: Skipped 19 previous similar messages [14340.769129] LNetError: 463694:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.146 [14340.784795] LNetError: 463694:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [14340.797865] LNetError: Protocol error connecting to 192.168.201.146@tcp124 at host 192.168.201.146:988: is it running a compatible version of Lustre? [14343.304880] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [14344.869995] LNetError: 461130:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.146 [14344.890270] LNetError: Protocol error connecting to 192.168.201.146@tcp124 at host 192.168.201.146:988: is it running a compatible version of Lustre? [14352.034471] LNetError: 461129:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.146 [14352.043687] LNetError: Protocol error connecting to 192.168.201.146@tcp124 at host 192.168.201.146:988: is it running a compatible version of Lustre? [14353.472854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14353.654770] Lustre: 463688:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14353.718575] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14353.722621] Lustre: Skipped 1 previous similar message [14353.795582] Lustre: lustre-MDT0001: new disk, initializing [14353.830200] Lustre: 464488:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.146@tcp: can't add NID 192.168.201.146@tcp130: rc = -75 [14353.838950] Lustre: 464488:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14353.929968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14353.981468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14353.992304] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14359.655400] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [14359.662389] Lustre: 464750:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.146@tcp: can't add NID 192.168.201.146@tcp130: rc = -75 [14359.681842] Lustre: 464750:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14368.428698] LNetError: 463699:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.146 [14368.441802] LNetError: Protocol error connecting to 192.168.201.146@tcp124 at host 192.168.201.146:988: is it running a compatible version of Lustre? [14371.098779] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14376.652155] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14383.232966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14383.543605] Lustre: lustre-OST0000: new disk, initializing [14383.548938] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14383.562630] Lustre: 465662:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14383.709334] Lustre: 465667:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.146@tcp: can't add NID 192.168.201.146@tcp130: rc = -75 [14383.725625] Lustre: 465667:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14383.791385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14384.740263] LNetError: 463698:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.146 [14384.747684] LNetError: 463698:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 12 previous similar messages [14384.757744] LNetError: Protocol error connecting to 192.168.201.146@tcp120 at host 192.168.201.146:988: is it running a compatible version of Lustre? [14384.766803] LNetError: Skipped 12 previous similar messages [14390.427199] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [14391.392901] Lustre: 466056:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.146@tcp: can't add NID 192.168.201.146@tcp130: rc = -75 [14391.405144] Lustre: 466056:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14391.490289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14391.509279] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14391.724354] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14398.183388] LNet: 1 local NIs in recovery (showing 1): 192.168.201.146@tcp124 [14401.260323] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14412.261483] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14412.280877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14412.283115] Lustre: Skipped 1 previous similar message [14412.297413] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14412.299145] Lustre: Skipped 1 previous similar message [14414.536548] Lustre: server umount lustre-OST0000 complete [14417.513528] LNetError: 461129:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.146 [14417.536896] LNetError: 461129:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 30 previous similar messages [14417.554233] LNetError: Protocol error connecting to 192.168.201.146@tcp129 at host 192.168.201.146:988: is it running a compatible version of Lustre? [14417.567632] LNetError: Skipped 30 previous similar messages [14420.450297] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14420.464123] LustreError: Skipped 1 previous similar message [14420.472459] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14420.500864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14422.505624] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14422.511231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14422.532048] Lustre: Skipped 1 previous similar message [14424.976537] Lustre: server umount lustre-MDT0000 complete [14429.095333] LustreError: 465661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451612 with bad export cookie 14991280890129686022 [14429.109308] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14429.497232] Lustre: server umount lustre-MDT0001 complete [14440.760528] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [14443.733477] Key type lgssc unregistered [14444.081757] LNet: 467572:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14444.094436] LNetError: 467572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14444.106341] LNet: Removed LNI 192.168.201.146@tcp [14445.152186] LNet: Removed LNI 192.168.201.146@tcp100 [14445.158466] LNetError: 467572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14445.166272] LNetError: 467572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14445.179465] LNet: Removed LNI 192.168.201.146@tcp101 [14446.240287] LNet: Removed LNI 192.168.201.146@tcp102 [14446.246666] LNetError: 467572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14446.253363] LNetError: 467572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14446.263683] LNet: Removed LNI 192.168.201.146@tcp103 [14447.328183] LNet: Removed LNI 192.168.201.146@tcp104 [14448.352212] LNet: Removed LNI 192.168.201.146@tcp105 [14448.375124] LNetError: 467572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14448.388354] LNetError: 467572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14448.401696] LNet: Removed LNI 192.168.201.146@tcp106 [14448.411626] LNet: Removed LNI 192.168.201.146@tcp107 [14449.440193] LNet: Removed LNI 192.168.201.146@tcp108 [14450.464371] LNet: Removed LNI 192.168.201.146@tcp109 [14451.488331] LNet: Removed LNI 192.168.201.146@tcp110 [14452.512436] LNet: Removed LNI 192.168.201.146@tcp111 [14452.525616] LNetError: 467572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14452.534970] LNetError: 467572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [14452.555990] LNet: Removed LNI 192.168.201.146@tcp112 [14453.600220] LNet: Removed LNI 192.168.201.146@tcp113 [14454.626322] LNet: Removed LNI 192.168.201.146@tcp114 [14455.648198] LNet: Removed LNI 192.168.201.146@tcp115 [14456.736282] LNet: Removed LNI 192.168.201.146@tcp116 [14457.760705] LNet: Removed LNI 192.168.201.146@tcp117 [14458.784154] LNet: Removed LNI 192.168.201.146@tcp118 [14459.808198] LNet: Removed LNI 192.168.201.146@tcp119 [14460.832135] LNet: Removed LNI 192.168.201.146@tcp120 [14460.858931] LNetError: 467572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14460.878605] LNetError: 467572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14460.892206] LNet: Removed LNI 192.168.201.146@tcp121 [14461.922326] LNet: Removed LNI 192.168.201.146@tcp122 [14462.944695] LNet: Removed LNI 192.168.201.146@tcp123 [14462.957045] LNet: Removed LNI 192.168.201.146@tcp124 [14464.032290] LNet: Removed LNI 192.168.201.146@tcp125 [14465.056329] LNet: Removed LNI 192.168.201.146@tcp126 [14466.080265] LNet: Removed LNI 192.168.201.146@tcp127 [14467.104528] LNet: Removed LNI 192.168.201.146@tcp128 [14468.128401] LNet: Removed LNI 192.168.201.146@tcp129 [14468.130228] LNet: Removed LNI 192.168.201.146@tcp130 [14468.138655] LNet: Removed LNI 192.168.201.146@tcp131 [14468.146711] LNet: Removed LNI 192.168.201.146@tcp132 [14468.153766] LNet: Removed LNI 192.168.201.146@tcp133 [14468.158266] LNet: Removed LNI 192.168.201.146@tcp134 [14468.175848] LNet: Removed LNI 192.168.201.146@tcp135 [14468.925059] Key type .llcrypt unregistered [14468.927122] Key type ._llcrypt unregistered [14494.024927] Key type ._llcrypt registered [14494.030648] Key type .llcrypt registered [14494.199871] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [14511.289772] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [14513.413435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14513.499792] alg: No test for adler32 (adler32-zlib) [14514.621758] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [14514.933937] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [14516.584257] Key type lgssc registered [14518.035559] Lustre: Echo OBD driver; http://www.lustre.org/ [14556.451403] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 12:08:58 (1788451738) [14576.585424] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [14579.704726] Key type lgssc unregistered [14580.051965] LNet: 471656:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14580.058939] LNetError: 471656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14580.075423] LNet: Removed LNI 192.168.201.146@tcp [14581.185978] Key type .llcrypt unregistered [14581.191456] Key type ._llcrypt unregistered [14600.001758] Key type ._llcrypt registered [14600.003996] Key type .llcrypt registered [14600.108685] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [14601.041919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14601.104623] alg: No test for adler32 (adler32-zlib) [14602.397502] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [14602.726726] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [14604.416242] Key type lgssc registered [14605.861251] Lustre: Echo OBD driver; http://www.lustre.org/ [14615.304164] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14615.341046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14616.564629] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14616.585289] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14616.688215] Lustre: lustre-MDT0000: new disk, initializing [14616.753466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14616.780303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14622.253552] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [14633.809279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14633.919628] Lustre: 472869:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14633.964671] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14633.968263] Lustre: Skipped 1 previous similar message [14634.028503] Lustre: lustre-MDT0001: new disk, initializing [14634.140066] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14634.187195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14634.196420] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14639.112485] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [14650.309939] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14657.010420] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14664.424836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14664.756968] Lustre: lustre-OST0000: new disk, initializing [14664.764109] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14664.776545] Lustre: 474839:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14664.873179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14669.948978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14669.960890] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14670.078779] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14671.866141] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [14682.180854] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14687.183302] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing lnet_if_list [14694.147859] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14694.206140] LNet: Added LNI 192.168.201.146@tcp10 [8/256/0/180] [14701.998173] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14702.074800] LNet: Added LNI 192.168.201.146@tcp11 [8/256/0/180] [14707.124865] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14707.162609] LNet: Added LNI 192.168.201.146@tcp12 [8/256/0/180] [14712.946754] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14712.973135] LNet: Added LNI 192.168.201.146@tcp13 [8/256/0/180] [14718.667380] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14718.721334] LNet: Added LNI 192.168.201.146@tcp14 [8/256/0/180] [14724.537910] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14724.563398] LNet: Added LNI 192.168.201.146@tcp15 [8/256/0/180] [14729.639449] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14729.661390] LNet: Added LNI 192.168.201.146@tcp16 [8/256/0/180] [14734.633312] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14734.657813] LNet: Added LNI 192.168.201.146@tcp17 [8/256/0/180] [14740.434142] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14740.460538] LNet: Added LNI 192.168.201.146@tcp18 [8/256/0/180] [14745.844589] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14745.908606] LNet: Added LNI 192.168.201.146@tcp19 [8/256/0/180] [14751.898173] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14751.911968] LNet: Added LNI 192.168.201.146@tcp20 [8/256/0/180] [14758.037163] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14758.067577] LNet: Added LNI 192.168.201.146@tcp21 [8/256/0/180] [14763.740576] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14763.779271] LNet: Added LNI 192.168.201.146@tcp22 [8/256/0/180] [14769.675891] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14769.710610] LNet: Added LNI 192.168.201.146@tcp23 [8/256/0/180] [14775.841934] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14775.864336] LNet: Added LNI 192.168.201.146@tcp24 [8/256/0/180] [14781.826884] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14781.849607] LNet: Added LNI 192.168.201.146@tcp25 [8/256/0/180] [14787.937940] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14787.968780] LNet: Added LNI 192.168.201.146@tcp26 [8/256/0/180] [14794.226503] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14794.255913] LNet: Added LNI 192.168.201.146@tcp27 [8/256/0/180] [14800.552597] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14800.580386] LNet: Added LNI 192.168.201.146@tcp28 [8/256/0/180] [14807.106121] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14807.125451] LNet: Added LNI 192.168.201.146@tcp29 [8/256/0/180] [14813.386021] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14813.418516] LNet: Added LNI 192.168.201.146@tcp30 [8/256/0/180] [14820.649926] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14820.713339] LNet: Added LNI 192.168.201.146@tcp31 [8/256/0/180] [14827.818814] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14827.845412] LNet: Added LNI 192.168.201.146@tcp32 [8/256/0/180] [14835.068682] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14835.120605] LNet: Added LNI 192.168.201.146@tcp33 [8/256/0/180] [14843.147770] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14843.193889] LNet: Added LNI 192.168.201.146@tcp34 [8/256/0/180] [14851.000602] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14851.025867] LNet: Added LNI 192.168.201.146@tcp35 [8/256/0/180] [14858.244575] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14858.299865] LNet: Added LNI 192.168.201.146@tcp36 [8/256/0/180] [14866.849678] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14866.870726] LNet: Added LNI 192.168.201.146@tcp37 [8/256/0/180] [14875.131626] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14875.158258] LNet: Added LNI 192.168.201.146@tcp38 [8/256/0/180] [14881.219414] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14881.244739] LNet: Added LNI 192.168.201.146@tcp39 [8/256/0/180] [14889.064657] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14889.090868] LNet: Added LNI 192.168.201.146@tcp40 [8/256/0/180] [14896.591558] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14896.619685] LNet: Added LNI 192.168.201.146@tcp41 [8/256/0/180] [14904.436623] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14904.464506] LNet: Added LNI 192.168.201.146@tcp42 [8/256/0/180] [14912.480879] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14912.508222] LNet: Added LNI 192.168.201.146@tcp43 [8/256/0/180] [14921.138638] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14921.180346] LNet: Added LNI 192.168.201.146@tcp44 [8/256/0/180] [14930.168789] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14930.207352] LNet: Added LNI 192.168.201.146@tcp45 [8/256/0/180] [14939.824675] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14939.856429] LNet: Added LNI 192.168.201.146@tcp46 [8/256/0/180] [14948.620089] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14948.647145] LNet: Added LNI 192.168.201.146@tcp47 [8/256/0/180] [14957.816531] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14957.866843] LNet: Added LNI 192.168.201.146@tcp48 [8/256/0/180] [14966.719363] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14966.739509] LNet: Added LNI 192.168.201.146@tcp49 [8/256/0/180] [14975.840421] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14975.865179] LNet: Added LNI 192.168.201.146@tcp50 [8/256/0/180] [14997.474256] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14997.486522] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14997.499965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15001.827823] Lustre: server umount lustre-MDT0000 complete [15005.731630] LustreError: 473665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452189 with bad export cookie 8845888730600124867 [15005.739180] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15006.041892] Lustre: server umount lustre-MDT0001 complete [15024.481844] Lustre: 472141:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788452191/real 1788452191] req@00000000234b63ee x1875327619079808/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788452207 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15024.512089] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [15024.516141] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15024.516149] Lustre: Skipped 2 previous similar messages [15026.664080] Lustre: 472140:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788452193/real 1788452193] req@000000007703c1b7 x1875327619079936/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788452209 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15029.728093] Lustre: 472139:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788452196/real 1788452196] req@00000000842bfd9b x1875327619080064/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788452212 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15032.672362] Lustre: 472141:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788452199/real 1788452199] req@000000002798cc8b x1875327619080320/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788452215 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15036.360038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15036.896343] Lustre: 472142:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788452204/real 1788452204] req@00000000a12afe56 x1875327619080448/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788452220 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15051.529848] LustreError: 474832:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15051.558472] LustreError: 474832:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [15051.614822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15053.669561] LustreError: 474833:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15053.687111] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15054.705404] LustreError: 474832:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15057.334893] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [15059.940135] LustreError: 474833:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15064.887798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15065.064779] LustreError: 474832:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15065.543233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15069.686731] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15070.929626] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [15078.731677] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15084.649889] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15105.511263] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15105.543763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15105.554201] Lustre: Skipped 3 previous similar messages [15110.639745] Lustre: server umount lustre-OST0000 complete [15115.745866] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15115.754367] Lustre: Skipped 1 previous similar message [15115.766137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15115.774485] Lustre: Skipped 1 previous similar message [15120.868765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15120.877360] Lustre: Skipped 1 previous similar message [15121.622750] Lustre: server umount lustre-MDT0000 complete [15124.962647] LustreError: 500013:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15124.977593] LustreError: 500013:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15125.566185] LustreError: 472862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452308 with bad export cookie 8845888730600127436 [15125.577346] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15125.579791] LustreError: 472862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [15125.864098] Lustre: server umount lustre-MDT0001 complete [15137.027149] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [15139.808386] Key type lgssc unregistered [15140.244899] LNet: 501883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15140.265106] LNetError: 501883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15140.272554] LNet: Removed LNI 192.168.201.146@tcp [15140.277239] LNet: Removed LNI 192.168.201.146@tcp10 [15140.280851] LNet: Removed LNI 192.168.201.146@tcp11 [15140.284570] LNet: Removed LNI 192.168.201.146@tcp12 [15140.287358] LNet: Removed LNI 192.168.201.146@tcp13 [15140.290153] LNet: Removed LNI 192.168.201.146@tcp14 [15140.293424] LNet: Removed LNI 192.168.201.146@tcp15 [15140.296262] LNet: Removed LNI 192.168.201.146@tcp16 [15140.299280] LNet: Removed LNI 192.168.201.146@tcp17 [15140.301764] LNet: Removed LNI 192.168.201.146@tcp18 [15140.304490] LNet: Removed LNI 192.168.201.146@tcp19 [15140.306996] LNet: Removed LNI 192.168.201.146@tcp20 [15140.309924] LNet: Removed LNI 192.168.201.146@tcp21 [15140.312825] LNet: Removed LNI 192.168.201.146@tcp22 [15140.315231] LNet: Removed LNI 192.168.201.146@tcp23 [15140.318878] LNet: Removed LNI 192.168.201.146@tcp24 [15140.322728] LNet: Removed LNI 192.168.201.146@tcp25 [15140.324974] LNet: Removed LNI 192.168.201.146@tcp26 [15140.327682] LNet: Removed LNI 192.168.201.146@tcp27 [15140.330530] LNet: Removed LNI 192.168.201.146@tcp28 [15140.333401] LNet: Removed LNI 192.168.201.146@tcp29 [15140.336543] LNet: Removed LNI 192.168.201.146@tcp30 [15140.339464] LNet: Removed LNI 192.168.201.146@tcp31 [15140.342522] LNet: Removed LNI 192.168.201.146@tcp32 [15140.345405] LNet: Removed LNI 192.168.201.146@tcp33 [15140.347646] LNet: Removed LNI 192.168.201.146@tcp34 [15140.349953] LNet: Removed LNI 192.168.201.146@tcp35 [15140.352507] LNet: Removed LNI 192.168.201.146@tcp36 [15140.355894] LNet: Removed LNI 192.168.201.146@tcp37 [15140.358202] LNet: Removed LNI 192.168.201.146@tcp38 [15140.361614] LNet: Removed LNI 192.168.201.146@tcp39 [15140.364459] LNet: Removed LNI 192.168.201.146@tcp40 [15140.367856] LNet: Removed LNI 192.168.201.146@tcp41 [15140.371321] LNet: Removed LNI 192.168.201.146@tcp42 [15140.373928] LNet: Removed LNI 192.168.201.146@tcp43 [15140.376502] LNet: Removed LNI 192.168.201.146@tcp44 [15140.379882] LNet: Removed LNI 192.168.201.146@tcp45 [15140.382815] LNet: Removed LNI 192.168.201.146@tcp46 [15140.384532] LNet: Removed LNI 192.168.201.146@tcp47 [15140.386539] LNet: Removed LNI 192.168.201.146@tcp48 [15140.388930] LNet: Removed LNI 192.168.201.146@tcp49 [15140.398928] LNet: Removed LNI 192.168.201.146@tcp50 [15141.081923] Key type .llcrypt unregistered [15141.084054] Key type ._llcrypt unregistered [15163.376458] Key type ._llcrypt registered [15163.378406] Key type .llcrypt registered [15163.557985] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [15183.340403] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [15184.792124] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15184.840960] alg: No test for adler32 (adler32-zlib) [15185.863929] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [15186.082720] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [15187.904660] Key type lgssc registered [15189.292948] Lustre: Echo OBD driver; http://www.lustre.org/ [15221.131330] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15221.158148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15222.467422] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15222.516979] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15222.572910] Lustre: lustre-MDT0000: new disk, initializing [15222.645094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15222.662146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15226.662174] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [15236.184112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15236.328901] Lustre: 505078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15236.386731] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15236.390582] Lustre: Skipped 1 previous similar message [15236.470949] Lustre: lustre-MDT0001: new disk, initializing [15236.584347] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15236.626119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15236.641731] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15240.459156] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [15249.241685] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15254.113051] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15262.222140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15262.375909] Lustre: *** cfs_fail_loc=90f, val=0*** [15262.437763] Lustre: lustre-OST0000: new disk, initializing [15262.443171] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15262.451810] Lustre: 507100:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15262.579716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15270.070257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15270.086053] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15270.297920] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15270.465740] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [15283.603152] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15295.974819] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15295.978394] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15295.992891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15296.003542] LustreError: Skipped 1 previous similar message [15296.024395] Lustre: Skipped 1 previous similar message [15297.790816] Lustre: server umount lustre-OST0000 complete [15303.138055] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15303.145637] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15303.176827] Lustre: Skipped 1 previous similar message [15303.190806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15306.209648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15306.211190] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15307.645352] Lustre: server umount lustre-MDT0000 complete [15311.071297] LustreError: 507099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452494 with bad export cookie 16846251425746176978 [15311.074084] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15311.084915] LustreError: 507099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15311.428540] Lustre: server umount lustre-MDT0001 complete [15323.219494] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [15325.946859] Key type lgssc unregistered [15326.371074] LNet: 509006:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15326.381684] LNetError: 509006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15327.400163] LNet: Removed LNI 192.168.201.146@tcp [15328.307256] Key type .llcrypt unregistered [15328.314886] Key type ._llcrypt unregistered [15349.261948] Key type ._llcrypt registered [15349.264930] Key type .llcrypt registered [15349.381521] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [15350.591510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15350.688906] alg: No test for adler32 (adler32-zlib) [15351.787485] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [15352.001499] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [15353.705710] Key type lgssc registered [15354.900769] Lustre: Echo OBD driver; http://www.lustre.org/ [15362.920798] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15362.947673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15364.427265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15369.108987] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [15373.807812] LustreError: 510272:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15373.844244] LustreError: 510272:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15378.917786] LustreError: 510273:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15379.253796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15379.597215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15384.083287] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [15391.825902] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15398.351403] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15405.272669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15405.687032] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15411.984700] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [15419.449602] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15432.344510] loop0: detected capacity change from 0 to 204800000 [15432.584869] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15432.596379] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15432.610091] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15438.685350] loop0: detected capacity change from 0 to 204800000 [15438.954826] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15438.980204] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15439.006091] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15443.935378] loop0: detected capacity change from 0 to 204800000 [15449.504948] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15449.685994] Lustre: fs15246-OST0000: new disk, initializing [15449.691676] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15449.699956] Lustre: 513865:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15449.760916] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15456.855214] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [15467.881871] loop1: detected capacity change from 0 to 204800000 [15473.482594] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15473.553603] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15473.602785] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15473.672560] Lustre: fs15246-MDT0000: new disk, initializing [15473.844147] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15473.867981] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15473.942628] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15474.979995] Lustre: 514986:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000006149b231 x1875328405387264/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788452674 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15478.827436] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [15480.035154] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15480.066968] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15480.101182] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15505.386421] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15505.431191] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15507.848878] Lustre: server umount fs15246-MDT0000 complete [15510.511890] LustreError: 510269:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15515.633548] LustreError: 510269:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15517.937956] Lustre: server umount fs15246-OST0000 complete [15530.977714] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15530.984190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15530.987589] Lustre: Skipped 1 previous similar message [15536.098148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15536.115766] Lustre: Skipped 1 previous similar message [15536.679681] Lustre: server umount lustre-OST0000 complete [15541.221400] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15541.229573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15541.241698] Lustre: Skipped 2 previous similar messages [15541.257793] Lustre: Skipped 1 previous similar message [15546.343964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15546.359026] Lustre: Skipped 1 previous similar message [15546.881575] Lustre: server umount lustre-MDT0000 complete [15550.311949] LustreError: 510252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452733 with bad export cookie 16640557976200484024 [15550.319072] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15550.321932] LustreError: 510252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15550.635614] Lustre: server umount lustre-MDT0001 complete [15563.389903] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [15566.156232] Key type lgssc unregistered [15566.453572] LNet: 518081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15566.457600] LNetError: 518081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15566.485336] LNet: Removed LNI 192.168.201.146@tcp [15567.287390] Key type .llcrypt unregistered [15567.289529] Key type ._llcrypt unregistered [15586.425461] Key type ._llcrypt registered [15586.426945] Key type .llcrypt registered [15586.547553] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [15587.965721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15588.147818] alg: No test for adler32 (adler32-zlib) [15589.305655] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [15589.690555] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [15591.418562] Key type lgssc registered [15592.828687] Lustre: Echo OBD driver; http://www.lustre.org/ [15638.965186] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [15650.645184] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [15696.470873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15696.519553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15697.978648] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15698.020538] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15698.198785] Lustre: lustre-MDT0000: new disk, initializing [15698.369948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15698.408134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15702.829502] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [15716.228475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15716.484268] Lustre: 522552:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15716.649512] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15716.661542] Lustre: Skipped 1 previous similar message [15716.907544] Lustre: lustre-MDT0001: new disk, initializing [15717.243322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15717.345936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15717.370666] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15722.252879] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [15734.719638] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15741.447528] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15749.553315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15749.840654] Lustre: lustre-OST0000: new disk, initializing [15749.844091] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15749.853184] Lustre: 524522:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15750.068531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15756.406841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15756.420674] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15756.649195] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15757.799758] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [15769.554396] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15776.034179] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15776.330885] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15782.458242] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15782.601657] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15787.489761] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15787.502349] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15787.513754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15787.549356] Lustre: Skipped 1 previous similar message [15787.572526] Lustre: Skipped 1 previous similar message [15791.140809] Lustre: server umount lustre-OST0000 complete [15797.737212] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15797.749138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15797.769798] Lustre: Skipped 1 previous similar message [15797.799826] Lustre: Skipped 1 previous similar message [15802.853589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15803.415160] Lustre: server umount lustre-MDT0000 complete [15808.260248] LustreError: 522543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452991 with bad export cookie 3260360878086676366 [15808.265468] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15808.279559] LustreError: 522543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15808.791132] Lustre: server umount lustre-MDT0001 complete [15839.420116] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [15854.944526] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [15893.174061] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15611 sec ========= 12:31:15 (1788453075) [15895.148119] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:31:17 (1788453077) === [15900.360607] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:31:22 (1788453082) === [15950.589055] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [15955.543319] Key type lgssc unregistered [15956.029121] LNet: 531191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15956.042494] LNetError: 531191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15956.075978] LNet: Removed LNI 192.168.201.146@tcp [15958.047195] Key type .llcrypt unregistered [15958.053669] Key type ._llcrypt unregistered