[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 497827620 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002381] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.006568] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.010149] LSM: Security Framework initializing [ 0.011058] Yama: becoming mindful. [ 0.012039] SELinux: Initializing. [ 0.013073] *** VALIDATE selinux *** [ 0.021686] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026312] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027157] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028117] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029131] *** VALIDATE tmpfs *** [ 0.031387] *** VALIDATE proc *** [ 0.032263] *** VALIDATE cgroup *** [ 0.034007] *** VALIDATE cgroup2 *** [ 0.035277] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037139] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039038] Spectre V2 : User space: Vulnerable [ 0.040011] Speculative Store Bypass: Vulnerable [ 0.043230] debug: unmapping init [mem 0xffffffff88c59000-0xffffffff88c60fff] [ 0.046000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046731] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047027] ... version: 2 [ 0.048013] ... bit width: 48 [ 0.049010] ... generic registers: 4 [ 0.050015] ... value mask: 0000ffffffffffff [ 0.051014] ... max period: 00007fffffffffff [ 0.052016] ... fixed-purpose events: 3 [ 0.053010] ... event mask: 000000070000000f [ 0.054314] rcu: Hierarchical SRCU implementation. [ 0.056559] smp: Bringing up secondary CPUs ... [ 0.057665] x86: Booting SMP configuration: [ 0.058034] .... node #0, CPUs: #1 #2 #3 [ 0.061213] smp: Brought up 1 node, 4 CPUs [ 0.063012] smpboot: Max logical packages: 1 [ 0.064016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.149606] node 0 deferred pages initialised in 83ms [ 0.153006] devtmpfs: initialized [ 0.154270] x86/mm: Memory block size: 128MB [ 0.156883] gcov: version magic: 0x41383552 [ 0.159248] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.163123] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.165308] pinctrl core: initialized pinctrl subsystem [ 0.167187] [ 0.167589] ************************************************************* [ 0.169015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.171014] ** ** [ 0.173014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.177020] ** ** [ 0.179013] ** This means that this kernel is built to expose internal ** [ 0.181012] ** IOMMU data structures, which may compromise security on ** [ 0.183014] ** your system. ** [ 0.185012] ** ** [ 0.186009] ** If you see this message and you are not debugging the ** [ 0.188014] ** kernel, report this immediately to your vendor! ** [ 0.190013] ** ** [ 0.192016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.194017] ************************************************************* [ 0.196656] NET: Registered protocol family 16 [ 0.198460] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.200059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.202059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.206017] cpuidle: using governor menu [ 0.208551] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.209579] PCI: Using configuration type 1 for base access [ 0.211139] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.219131] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.221037] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.225071] cryptd: max_cpu_qlen set to 1000 [ 0.228270] ACPI: Added _OSI(Module Device) [ 0.230019] ACPI: Added _OSI(Processor Device) [ 0.231013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.232010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.237000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.241540] ACPI: Interpreter enabled [ 0.243064] ACPI: PM: (supports S0 S3 S4 S5) [ 0.244013] ACPI: Using IOAPIC for interrupt routing [ 0.246113] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.249630] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.260826] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.263048] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.266022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.268080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.273249] acpiphp: Slot [2] registered [ 0.274079] acpiphp: Slot [5] registered [ 0.275074] acpiphp: Slot [6] registered [ 0.276103] acpiphp: Slot [7] registered [ 0.278104] acpiphp: Slot [8] registered [ 0.279129] acpiphp: Slot [9] registered [ 0.280121] acpiphp: Slot [10] registered [ 0.282114] acpiphp: Slot [3] registered [ 0.283093] acpiphp: Slot [4] registered [ 0.285107] acpiphp: Slot [11] registered [ 0.286102] acpiphp: Slot [12] registered [ 0.287111] acpiphp: Slot [13] registered [ 0.289126] acpiphp: Slot [14] registered [ 0.290133] acpiphp: Slot [15] registered [ 0.292111] acpiphp: Slot [16] registered [ 0.293105] acpiphp: Slot [17] registered [ 0.295109] acpiphp: Slot [18] registered [ 0.296128] acpiphp: Slot [19] registered [ 0.298117] acpiphp: Slot [20] registered [ 0.299103] acpiphp: Slot [21] registered [ 0.301087] acpiphp: Slot [22] registered [ 0.302073] acpiphp: Slot [23] registered [ 0.303059] acpiphp: Slot [24] registered [ 0.304066] acpiphp: Slot [25] registered [ 0.305067] acpiphp: Slot [26] registered [ 0.306067] acpiphp: Slot [27] registered [ 0.307069] acpiphp: Slot [28] registered [ 0.308095] acpiphp: Slot [29] registered [ 0.309038] acpiphp: Slot [30] registered [ 0.310066] acpiphp: Slot [31] registered [ 0.311067] PCI host bridge to bus 0000:00 [ 0.311999] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.313016] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.315017] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.317017] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.318015] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.320029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.322157] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.323736] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.327203] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.336716] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.342487] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.345026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.347013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.350018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.351467] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.354826] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.357038] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.359731] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.363631] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.373016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.377017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.381739] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.392025] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.399018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.424021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.437475] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.462016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.469015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.487017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.498083] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.519022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.537019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.569029] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.586740] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.600018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.610018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.635016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.645617] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.652017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.659018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.673071] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.688731] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.700050] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.707020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.721020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.731173] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.733384] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.735316] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.737272] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.738136] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.742150] iommu: Default domain type: Passthrough [ 0.743313] SCSI subsystem initialized [ 0.744105] ACPI: bus type USB registered [ 0.744952] usbcore: registered new interface driver usbfs [ 0.746057] usbcore: registered new interface driver hub [ 0.747085] usbcore: registered new device driver usb [ 0.748106] pps_core: LinuxPPS API ver. 1 registered [ 0.749006] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.751057] PTP clock support registered [ 0.753129] EDAC MC: Ver: 3.0.0 [ 0.754319] PCI: Using ACPI for IRQ routing [ 0.755655] NetLabel: Initializing [ 0.757014] NetLabel: domain hash size = 128 [ 0.757952] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.759063] NetLabel: unlabeled traffic allowed by default [ 0.761118] vgaarb: loaded [ 0.762244] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.763011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.768005] clocksource: Switched to clocksource kvm-clock [ 0.869603] VFS: Disk quotas dquot_6.6.0 [ 0.871302] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.873101] *** VALIDATE ramfs *** [ 0.874349] *** VALIDATE hugetlbfs *** [ 0.875679] pnp: PnP ACPI init [ 0.877266] pnp: PnP ACPI: found 6 devices [ 0.891701] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.894149] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.895558] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.897063] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.898687] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.900386] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.902299] NET: Registered protocol family 2 [ 0.904116] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.907555] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.910612] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.914709] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.917578] TCP: Hash tables configured (established 65536 bind 65536) [ 0.920018] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.922113] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.924521] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.926271] NET: Registered protocol family 1 [ 0.928321] RPC: Registered named UNIX socket transport module. [ 0.930772] RPC: Registered udp transport module. [ 0.932771] RPC: Registered tcp transport module. [ 0.934709] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.937290] NET: Registered protocol family 44 [ 0.938214] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.940143] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.942116] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.944206] PCI: CLS 0 bytes, default 64 [ 0.945526] Unpacking initramfs... [ 2.349408] debug: unmapping init [mem 0xffff90237cc54000-0xffff90237ffbffff] [ 2.353862] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.356625] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.360344] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.861406] Initialise system trusted keyrings [ 2.862654] Key type blacklist registered [ 2.864332] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.873818] zbud: loaded [ 2.876988] *** VALIDATE nfs *** [ 2.878316] *** VALIDATE nfs4 *** [ 2.880041] pstore: using deflate compression [ 2.884872] Platform Keyring initialized [ 3.001070] NET: Registered protocol family 38 [ 3.002290] Key type asymmetric registered [ 3.003517] Asymmetric key parser 'x509' registered [ 3.005140] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.007590] io scheduler mq-deadline registered [ 3.009384] io scheduler kyber registered [ 3.011230] io scheduler bfq registered [ 3.013299] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.016445] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.019353] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.022243] ACPI: Power Button [PWRF] [ 3.027747] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.034212] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.049106] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.056903] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.088273] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.119910] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.151248] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.156792] Non-volatile memory driver v1.3 [ 3.158457] Linux agpgart interface v0.103 [ 3.187043] virtio_blk virtio1: [vda] 146008 512-byte logical blocks (74.8 MB/71.3 MiB) [ 3.189795] vda: detected capacity change from 0 to 74756096 [ 3.216890] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.219695] vdb: detected capacity change from 0 to 1073741824 [ 3.234138] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.236962] vdc: detected capacity change from 0 to 2621440000 [ 3.250728] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.252926] vdd: detected capacity change from 0 to 2621440000 [ 3.269190] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.271790] vde: detected capacity change from 0 to 4294967296 [ 3.284321] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.286391] vdf: detected capacity change from 0 to 4294967296 [ 3.295976] libphy: Fixed MDIO Bus: probed [ 3.303288] usbcore: registered new interface driver usbserial_generic [ 3.307487] usbserial: USB Serial support registered for generic [ 3.309661] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.313580] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.314801] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.316566] mousedev: PS/2 mouse device common for all mice [ 3.319874] rtc_cmos 00:05: RTC can wake from S4 [ 3.325169] rtc_cmos 00:05: registered as rtc0 [ 3.327890] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.327899] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.334854] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.334942] intel_pstate: CPU model not supported [ 3.340636] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.344420] hid: raw HID events driver (C) Jiri Kosina [ 3.347184] usbcore: registered new interface driver usbhid [ 3.349872] usbhid: USB HID core driver [ 3.351397] drop_monitor: Initializing network drop monitor service [ 3.353729] Initializing XFRM netlink socket [ 3.355702] NET: Registered protocol family 10 [ 3.358729] Segment Routing with IPv6 [ 3.359985] NET: Registered protocol family 17 [ 3.361965] mpls_gso: MPLS GSO support [ 3.367317] RAS: Correctable Errors collector initialized. [ 3.369402] AVX version of gcm_enc/dec engaged. [ 3.370959] AES CTR mode by8 optimization enabled [ 3.436625] sched_clock: Marking stable (3436608561, 0)->(4330549870, -893941309) [ 3.439099] registered taskstats version 1 [ 3.440293] Loading compiled-in X.509 certificates [ 3.441457] zswap: loaded using pool lzo/zbud [ 3.469847] Key type big_key registered [ 3.482967] Key type encrypted registered [ 3.484655] ima: No TPM chip found, activating TPM-bypass! [ 3.486720] ima: Allocated hash algorithm: sha1 [ 3.488598] ima: No architecture policies found [ 3.490530] evm: Initialising EVM extended attributes: [ 3.492614] evm: security.selinux [ 3.493957] evm: security.ima [ 3.495359] evm: security.capability [ 3.497053] evm: HMAC attrs: 0x1 [ 3.500835] rtc_cmos 00:05: setting system clock to 2026-08-19 04:51:32 UTC (1787115092) [ 3.506992] debug: unmapping init [mem 0xffffffff89c03000-0xffffffff89dfffff] [ 3.509206] debug: unmapping init [mem 0xffffffff88982000-0xffffffff88c58fff] [ 3.517064] Write protecting the kernel read-only data: 28672k [ 3.519079] debug: unmapping init [mem 0xffffffff87003000-0xffffffff871fffff] [ 3.520741] debug: unmapping init [mem 0xffffffff87914000-0xffffffff879fffff] [ 3.549331] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.555586] systemd[1]: Detected virtualization kvm. [ 3.557603] systemd[1]: Detected architecture x86-64. [ 3.559664] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.587942] systemd[1]: No hostname configured. [ 3.589828] systemd[1]: Set hostname to . [ 3.592331] random: systemd: uninitialized urandom read (16 bytes read) [ 3.594213] systemd[1]: Initializing machine ID from random generator. [ 3.740368] random: systemd: uninitialized urandom read (16 bytes read) [ 3.743275] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.747891] random: systemd: uninitialized urandom read (16 bytes read) [ 3.750940] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.755599] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. Starting Journal Service... [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.370679] device-mapper: uevent: version 1.0.3 [ 4.373072] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.143126] virtio_net virtio0 ens2: renamed from eth0 [ 5.154174] random: fast init done [ 5.238282] scsi host0: ata_piix [ 5.249156] scsi host1: ata_piix [ 5.250854] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.253559] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.088746] dracut-initqueue[583]: RTNETLINK answers: File exists [ 10.003094] random: crng init done [ 10.004425] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.559038] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.785799] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.065281] SELinux: Disabled at runtime. [ 12.125550] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.134648] systemd[1]: Detected virtualization kvm. [ 12.136460] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.722949] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.726279] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.730926] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.734948] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.738396] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.746878] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.754814] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. Mounting Huge Pages File System... Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd Root File System. [ 12.921091] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.271568] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.527291] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.555041] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.715175] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.725159] EDAC sbridge: Ver: 1.1.2 [ 15.478786] Key type dns_resolver registered [ 15.784574] NFS: Registering the id_resolver key type [ 15.786701] Key type id_resolver registered [ 15.788370] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Mark the need to relabel after reboot. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg126-server login: [ 74.715301] hrtimer: interrupt took 4291725 ns [ 76.529650] libcfs: loading out-of-tree module taints kernel. [ 76.571956] Key type ._llcrypt registered [ 76.573767] Key type .llcrypt registered [ 76.656969] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 100.335869] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 102.747356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 102.769918] alg: No test for adler32 (adler32-zlib) [ 104.329382] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 105.589730] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 107.503187] Key type lgssc registered [ 109.553878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 129.959683] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 172.445900] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 185.602889] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 185.637909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 186.950050] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 186.999111] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 187.102710] Lustre: lustre-MDT0000: new disk, initializing [ 187.160633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 187.195042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 192.229863] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 207.683715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 207.793494] Lustre: 6517:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 207.838341] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 207.847362] Lustre: Skipped 1 previous similar message [ 207.962427] Lustre: lustre-MDT0001: new disk, initializing [ 208.027678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 208.052723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 208.106239] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 212.982381] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 218.670918] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 231.812763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 232.348656] Lustre: lustre-OST0000: new disk, initializing [ 232.350968] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 232.371150] Lustre: 8457:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 232.494839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 240.158285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 240.173293] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 240.259432] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 241.528634] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 258.924935] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 259.113550] Lustre: lustre-OST0001: new disk, initializing [ 259.119323] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 259.126216] Lustre: 9531:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 259.204571] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 266.351613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 266.374968] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 266.434748] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 266.896725] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 281.301916] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 289.354832] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 296.325369] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing check_logdir /tmp/testlogs/ [ 302.405778] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing yml_node [ 307.099739] Lustre: DEBUG MARKER: Client: 2.17.56.4 [ 310.108738] Lustre: DEBUG MARKER: MDS: 2.17.56.4 [ 312.767883] Lustre: DEBUG MARKER: OSS: 2.17.56.4 [ 314.798484] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Aug 19 00:56:41 EDT 2026 [ 332.719905] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 334.570816] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 353.766575] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 353.769158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 353.790264] Lustre: Skipped 3 previous similar messages [ 353.808733] Lustre: Skipped 3 previous similar messages [ 358.881439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 358.886440] Lustre: Skipped 3 previous similar messages [ 359.151598] Lustre: server umount lustre-MDT0000 complete [ 362.827945] LustreError: 6511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787115451 with bad export cookie 5921647044649849913 [ 362.829285] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 362.836115] LustreError: 6511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 364.002411] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 364.002864] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 364.019251] Lustre: Skipped 1 previous similar message [ 367.008315] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 367.028768] Lustre: Skipped 2 previous similar messages [ 372.128315] LustreError: 6526:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 372.129165] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 372.153518] LustreError: 6526:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 372.159606] Lustre: Skipped 1 previous similar message [ 377.311193] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 377.311989] LustreError: 8466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 377.344301] LustreError: 8466:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 377.641534] Lustre: server umount lustre-MDT0001 complete [ 381.869516] Lustre: server umount lustre-OST0000 complete [ 385.604043] Lustre: server umount lustre-OST0001 complete [ 391.792754] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 401.572572] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 442.784894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 443.016312] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 443.045619] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 443.134387] Lustre: lustre-MDT0000: new disk, initializing [ 443.304054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 443.337141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 447.848135] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 460.956403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 461.230233] Lustre: 15089:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 461.327336] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 461.331573] Lustre: Skipped 1 previous similar message [ 461.446991] Lustre: lustre-MDT0001: new disk, initializing [ 461.608602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 461.700757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 461.735590] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 467.803828] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 481.299380] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 488.310797] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 496.474226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 496.803644] Lustre: lustre-OST0000: new disk, initializing [ 496.817097] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 496.828409] Lustre: 17056:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 496.996276] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 498.329420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 498.337156] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 498.569977] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 503.716636] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 513.206824] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 518.519608] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 518.772135] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 524.428347] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 524.681426] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 529.379428] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 529.388604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 529.414312] Lustre: Skipped 1 previous similar message [ 529.420538] Lustre: Skipped 2 previous similar messages [ 541.152248] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 541.378251] Lustre: server umount lustre-OST0000 complete [ 548.832955] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 548.847492] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 548.882742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 548.890631] Lustre: Skipped 5 previous similar messages [ 552.961335] Lustre: server umount lustre-MDT0000 complete [ 554.977903] LustreError: 15099:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 555.021714] LustreError: 15099:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 557.790291] LustreError: 17055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787115646 with bad export cookie 5921647044649851432 [ 557.799056] LustreError: 17055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 557.799789] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 558.088793] Lustre: server umount lustre-MDT0001 complete [ 560.688558] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 01:00:48 (1787115648) [ 567.012200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 567.470791] LustreError: 19323:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 567.542654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 572.279340] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 572.897118] LustreError: 19324:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 580.872082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 587.292707] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 597.116639] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 603.897947] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 611.278826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 621.460327] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 630.084925] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 639.458894] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 639.475140] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 639.492157] Lustre: Skipped 1 previous similar message [ 639.509266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 639.511915] Lustre: Skipped 2 previous similar messages [ 642.528462] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 644.833692] Lustre: server umount lustre-OST0000 complete [ 652.768903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 652.769893] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 652.797453] Lustre: Skipped 2 previous similar messages [ 656.153928] Lustre: server umount lustre-MDT0000 complete [ 657.888838] LustreError: 19320:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 657.909349] LustreError: 19320:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 660.469751] LustreError: 20072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787115749 with bad export cookie 5921647044649852314 [ 660.484274] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 660.486461] LustreError: 20072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 661.006662] Lustre: server umount lustre-MDT0001 complete [ 673.590566] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 676.435195] Key type lgssc unregistered [ 676.704183] LNet: 22870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 676.711711] LNetError: 22870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 677.746466] LNet: Removed LNI 192.168.201.126@tcp [ 678.773344] Key type .llcrypt unregistered [ 678.776630] Key type ._llcrypt unregistered [ 703.634440] Key type ._llcrypt registered [ 703.639093] Key type .llcrypt registered [ 703.754554] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 705.157548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 705.288883] alg: No test for adler32 (adler32-zlib) [ 706.300394] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 706.500800] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 708.159249] Key type lgssc registered [ 709.598879] Lustre: Echo OBD driver; http://www.lustre.org/ [ 718.523684] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 718.567484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 720.195517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 725.114666] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 730.595599] LustreError: 24284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 730.613597] LustreError: 24284:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 734.455597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 734.801805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 740.075461] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 748.479902] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 754.317623] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 762.527090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 763.160461] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 765.188391] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 770.717784] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 779.131415] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 796.136585] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 796.145607] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 796.167631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 798.362603] Lustre: server umount lustre-OST0000 complete [ 805.356389] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 805.362688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 805.379358] Lustre: Skipped 1 previous similar message [ 805.391107] Lustre: Skipped 1 previous similar message [ 808.869899] Lustre: server umount lustre-MDT0000 complete [ 810.469777] LustreError: 25011:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 810.507676] LustreError: 25011:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 812.261612] LustreError: 24264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787115901 with bad export cookie 7450766924636483801 [ 812.270645] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 812.278339] LustreError: 24264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 822.861869] Lustre: server umount lustre-MDT0001 complete [ 833.181403] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 835.936316] Key type lgssc unregistered [ 836.203571] LNet: 28259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 836.207128] LNetError: 28259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 836.219869] LNet: Removed LNI 192.168.201.126@tcp [ 837.040344] Key type .llcrypt unregistered [ 837.043168] Key type ._llcrypt unregistered [ 863.315582] Key type ._llcrypt registered [ 863.317486] Key type .llcrypt registered [ 863.407399] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 864.298558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 864.376224] alg: No test for adler32 (adler32-zlib) [ 865.461211] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 865.644863] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 867.295162] Key type lgssc registered [ 868.724726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 878.292790] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 878.305519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 879.706265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 884.354373] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 889.843884] LustreError: 29675:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 889.865082] LustreError: 29675:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 893.392298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 893.922076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 899.035378] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 907.412384] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 915.364632] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 930.687429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 931.141357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 938.078723] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 945.468389] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 949.627906] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 958.948074] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 958.955477] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 958.960972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 959.970329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 959.978103] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 959.989225] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 963.174481] Lustre: server umount lustre-OST0000 complete [ 970.210802] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 970.218097] Lustre: Skipped 1 previous similar message [ 970.219685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 970.226039] Lustre: Skipped 2 previous similar messages [ 974.207071] Lustre: server umount lustre-MDT0000 complete [ 975.330970] LustreError: 29674:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 975.355634] LustreError: 29674:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 978.375315] LustreError: 29656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787116067 with bad export cookie 8900403315751663107 [ 978.378153] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 978.385480] LustreError: 29656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 978.670520] Lustre: server umount lustre-MDT0001 complete [ 989.134553] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 992.001056] Key type lgssc unregistered [ 992.389483] LNet: 33556:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 992.398462] LNetError: 33556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 992.424836] LNet: Removed LNI 192.168.201.126@tcp [ 993.308918] Key type .llcrypt unregistered [ 993.314334] Key type ._llcrypt unregistered [ 1015.665544] Key type ._llcrypt registered [ 1015.667405] Key type .llcrypt registered [ 1015.772376] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 1017.031476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1017.078343] alg: No test for adler32 (adler32-zlib) [ 1018.231287] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 1018.485408] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 1020.199202] Key type lgssc registered [ 1021.514032] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1029.999068] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1030.015702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1031.486390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1035.881700] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1041.891284] LustreError: 34970:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1041.911292] LustreError: 34970:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1044.840704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1045.220852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1049.980625] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1058.329363] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1064.630677] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1073.016337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1073.519456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1076.598214] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1083.099364] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1093.770486] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1107.429264] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1107.439553] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1107.462194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1107.464354] Lustre: Skipped 1 previous similar message [ 1110.983911] Lustre: server umount lustre-OST0000 complete [ 1117.152651] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1117.174416] LustreError: Skipped 1 previous similar message [ 1117.180845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1117.194148] Lustre: Skipped 1 previous similar message [ 1117.204073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1121.638110] LustreError: 37751:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x70616d65646f6e:0x4:0x0].0x0 (00000000c29229ee) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 1122.787152] LustreError: 34970:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1122.819571] LustreError: 34970:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1123.716252] Lustre: server umount lustre-MDT0000 complete [ 1127.908776] LustreError: 34971:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1127.940883] LustreError: 34971:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1128.456026] LustreError: 34949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787116217 with bad export cookie 17415202702013299022 [ 1128.463674] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1128.500838] LustreError: 34949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1128.838210] Lustre: server umount lustre-MDT0001 complete [ 1141.786362] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 1144.896478] Key type lgssc unregistered [ 1145.319358] LNet: 38504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1145.333863] LNetError: 38504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1145.364661] LNet: Removed LNI 192.168.201.126@tcp [ 1146.602121] Key type .llcrypt unregistered [ 1146.604834] Key type ._llcrypt unregistered [ 1170.804460] Key type ._llcrypt registered [ 1170.807206] Key type .llcrypt registered [ 1170.950987] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 1172.309282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1172.356827] alg: No test for adler32 (adler32-zlib) [ 1173.419153] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 1173.615554] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 1175.295203] Key type lgssc registered [ 1176.281746] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1184.520730] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1184.549725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1186.162324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1191.662779] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1196.523352] LustreError: 39920:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1196.555498] LustreError: 39920:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1201.632235] LustreError: 39919:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1202.252310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1202.688320] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1208.380037] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1217.212106] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1223.701856] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1230.639565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1230.992569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1232.049130] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1237.921562] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1246.274641] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1254.236288] Lustre: lustre-OST0000: Not available for connect from 192.168.201.26@tcp (stopping) [ 1257.954715] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1257.964678] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1257.996204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1257.996211] Lustre: Skipped 1 previous similar message [ 1258.015790] LustreError: Skipped 1 previous similar message [ 1259.313195] Lustre: lustre-OST0000: Not available for connect from 192.168.201.26@tcp (stopping) [ 1263.080405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1263.095826] Lustre: Skipped 1 previous similar message [ 1267.167567] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1267.438495] Lustre: server umount lustre-OST0000 complete [ 1288.036691] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1288.044481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1288.064474] Lustre: Skipped 2 previous similar messages [ 1288.068268] Lustre: Skipped 1 previous similar message [ 1293.287823] LustreError: 40648:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1293.309280] LustreError: 40648:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1293.355786] Lustre: server umount lustre-MDT0000 complete [ 1296.600331] LustreError: 41763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787116385 with bad export cookie 17055837527854841234 [ 1296.603980] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1296.609874] LustreError: 41763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1303.083145] Lustre: server umount lustre-MDT0001 complete [ 1315.732990] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 1318.419608] Key type lgssc unregistered [ 1318.630474] LNet: 43629:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1318.640589] LNetError: 43629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1318.656071] LNet: Removed LNI 192.168.201.126@tcp [ 1319.470343] Key type .llcrypt unregistered [ 1319.474684] Key type ._llcrypt unregistered [ 1343.337416] Key type ._llcrypt registered [ 1343.344701] Key type .llcrypt registered [ 1343.552201] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 1344.868657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1344.901875] alg: No test for adler32 (adler32-zlib) [ 1345.981089] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 1346.289339] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 1348.031355] Key type lgssc registered [ 1349.565824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1358.126243] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1358.142781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1359.530827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1364.172691] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1369.059629] LustreError: 45043:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1369.074530] LustreError: 45043:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1372.554888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1372.922558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1377.767201] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1384.700027] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1390.475947] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1397.141747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1397.635450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1403.898364] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1404.804137] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1410.485357] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1417.549500] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.26@tcp (stopping) [ 1418.135375] Lustre: server umount lustre-MDT0000 complete [ 1419.232289] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1419.242054] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1419.262964] LustreError: 45042:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1420.262078] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1420.263103] LustreError: 45043:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1420.295395] LustreError: 45043:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1421.562901] LustreError: 46886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787116510 with bad export cookie 8729114509029900328 [ 1421.573720] LustreError: 46886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1421.577128] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1422.641294] LustreError: 45038:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1422.666364] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.26@tcp (stopping) [ 1425.378104] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1425.391702] Lustre: Skipped 1 previous similar message [ 1425.396161] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1427.765730] LustreError: 45038:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1427.767109] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.26@tcp (stopping) [ 1427.793911] LustreError: 45038:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1427.818391] Lustre: Skipped 1 previous similar message [ 1427.968423] Lustre: server umount lustre-MDT0001 complete [ 1448.731411] Lustre: server umount lustre-OST0000 complete [ 1465.871347] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 1468.697230] Key type lgssc unregistered [ 1469.070395] LNet: 48903:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1469.078530] LNetError: 48903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1469.127309] LNet: Removed LNI 192.168.201.126@tcp [ 1470.124929] Key type .llcrypt unregistered [ 1470.126981] Key type ._llcrypt unregistered [ 1496.978897] Key type ._llcrypt registered [ 1496.981485] Key type .llcrypt registered [ 1497.134459] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 1498.496938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1498.540339] alg: No test for adler32 (adler32-zlib) [ 1499.620634] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 1499.798715] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 1501.431155] Key type lgssc registered [ 1502.701088] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1510.392865] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1510.432178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1526.050543] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1531.167340] LustreError: 50291:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.126@tcp: failed processing log, type 4: rc = -110 [ 1557.855424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1564.614629] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1572.544484] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1737.816849] Lustre: server umount lustre-OST0000 complete [ 1754.553971] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 1757.128070] Key type lgssc unregistered [ 1757.392724] LNet: 51967:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1757.404285] LNetError: 51967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1757.416947] LNet: Removed LNI 192.168.201.126@tcp [ 1758.100579] Key type .llcrypt unregistered [ 1758.102979] Key type ._llcrypt unregistered [ 1781.196152] Key type ._llcrypt registered [ 1781.198459] Key type .llcrypt registered [ 1781.285669] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 1782.396885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1782.421409] alg: No test for adler32 (adler32-zlib) [ 1783.433592] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 1783.750093] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 1785.487250] Key type lgssc registered [ 1786.664797] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1794.149858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1794.176883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1795.713185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1800.352917] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1805.286900] LustreError: 53382:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1805.306890] LustreError: 53382:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1808.667174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1809.164286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1814.336550] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1824.133968] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1830.043673] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1837.195064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1837.877157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1840.947153] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1845.488907] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1856.266903] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1861.605082] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1861.609753] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1861.617372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1864.675301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1864.677375] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1869.794408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1869.809511] Lustre: Skipped 1 previous similar message [ 1873.887567] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1874.005571] Lustre: server umount lustre-OST0000 complete [ 1880.032561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1880.058264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1880.076258] Lustre: Skipped 2 previous similar messages [ 1884.715526] Lustre: server umount lustre-MDT0000 complete [ 1885.157091] LustreError: 53382:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1885.190434] LustreError: 53382:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1888.096794] LustreError: 53362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787116977 with bad export cookie 1525875988046129122 [ 1888.109214] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1888.309193] Lustre: server umount lustre-MDT0001 complete [ 1903.124491] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 1906.555058] Key type lgssc unregistered [ 1906.987602] LNet: 56921:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1906.998259] LNetError: 56921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1907.030158] LNet: Removed LNI 192.168.201.126@tcp [ 1908.233279] Key type .llcrypt unregistered [ 1908.235662] Key type ._llcrypt unregistered [ 1938.523564] Key type ._llcrypt registered [ 1938.525672] Key type .llcrypt registered [ 1938.608665] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 1939.577948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1939.726770] alg: No test for adler32 (adler32-zlib) [ 1940.813872] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 1941.174398] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 1942.855784] Key type lgssc registered [ 1944.143619] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1952.791431] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1952.807307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1968.287900] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1973.407325] LustreError: 58310:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.126@tcp: failed processing log, type 4: rc = -110 [ 2000.031586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2007.104314] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2015.080991] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2021.344548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2021.817588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2021.904751] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2026.331786] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2026.981809] LustreError: 58336:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2027.002637] LustreError: 58336:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2032.099923] LustreError: 58335:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2034.147431] LustreError: 58336:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2034.163259] LustreError: 58336:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2035.123935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2035.636113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2040.262966] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2047.003864] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2052.261726] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2056.169920] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2056.171721] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2056.173475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2056.182067] Lustre: Skipped 1 previous similar message [ 2060.734760] Lustre: server umount lustre-OST0000 complete [ 2086.882370] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2086.885318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2086.901918] Lustre: Skipped 1 previous similar message [ 2086.930736] Lustre: Skipped 2 previous similar messages [ 2090.181418] Lustre: server umount lustre-MDT0000 complete [ 2092.002035] LustreError: 60951:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2092.027718] LustreError: 60951:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2095.030681] LustreError: 58316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787117184 with bad export cookie 10268400322435664281 [ 2095.036912] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2095.069071] LustreError: 58316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2101.751417] Lustre: server umount lustre-MDT0001 complete [ 2113.397918] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 2117.067755] Key type lgssc unregistered [ 2117.504898] LNet: 62054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2117.509775] LNetError: 62054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2117.524188] LNet: Removed LNI 192.168.201.126@tcp [ 2118.797091] Key type .llcrypt unregistered [ 2118.799462] Key type ._llcrypt unregistered [ 2143.487876] Key type ._llcrypt registered [ 2143.490029] Key type .llcrypt registered [ 2143.604198] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 2144.820474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2144.982704] alg: No test for adler32 (adler32-zlib) [ 2146.116944] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 2146.392909] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 2148.039333] Key type lgssc registered [ 2149.556702] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2157.403371] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2157.436458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2158.793130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2164.484533] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2168.292190] LustreError: 63467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2168.321538] LustreError: 63467:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2173.437536] LustreError: 63468:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2174.783847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2175.408488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2180.687793] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2188.792858] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2194.983285] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2201.222029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2201.482618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2203.588439] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2207.492728] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2215.053830] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2234.338307] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2234.342134] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2234.352537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2236.386650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2236.403539] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2236.427388] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2239.993523] Lustre: server umount lustre-OST0000 complete [ 2246.636479] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2246.683188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2246.692161] Lustre: Skipped 2 previous similar messages [ 2251.310264] Lustre: server umount lustre-MDT0000 complete [ 2251.744978] LustreError: 63463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2251.766790] LustreError: 63463:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2255.237321] LustreError: 63449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787117344 with bad export cookie 9664979644280710456 [ 2255.246720] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2255.250312] LustreError: 63449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2255.590560] Lustre: server umount lustre-MDT0001 complete [ 2266.499958] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 2269.458392] Key type lgssc unregistered [ 2269.723960] LNet: 67027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2269.729351] LNetError: 67027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2269.740572] LNet: Removed LNI 192.168.201.126@tcp [ 2270.500380] Key type .llcrypt unregistered [ 2270.503518] Key type ._llcrypt unregistered [ 2298.116949] Key type ._llcrypt registered [ 2298.119755] Key type .llcrypt registered [ 2298.246706] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 2299.262116] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2299.287755] alg: No test for adler32 (adler32-zlib) [ 2300.381520] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 2300.542679] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 2302.207461] Key type lgssc registered [ 2303.286505] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2312.260724] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 01:29:59 (1787117399) [ 2318.912833] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2318.952860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2320.366883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2324.394861] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2330.593699] LustreError: 68732:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2330.650249] LustreError: 68732:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2333.014244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2333.378443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2338.907480] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2347.837451] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2352.987723] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2361.356295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2361.738494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2365.902085] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2369.596606] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2378.503738] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2385.876151] Lustre: Failing over lustre-MDT0000 [ 2386.250835] LustreError: 70772:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2386.269302] Lustre: server umount lustre-MDT0000 complete [ 2388.454883] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2388.458319] LustreError: 68728:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2388.470734] Lustre: Skipped 1 previous similar message [ 2388.518328] LustreError: 68728:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2391.345149] LustreError: 68727:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2396.072762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2396.163081] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2396.263863] Lustre: *** cfs_fail_loc=135, val=0*** [ 2396.323857] LustreError: 71816:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2396.328867] LustreError: 71816:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.126@tcp: cfg command failed: rc = -2 [ 2396.336287] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2396.336287] [ 2396.344138] LustreError: MGC192.168.201.126@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2396.360366] LustreError: 71804:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2396.369155] LustreError: 71804:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2396.373496] LustreError: 71804:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2396.382845] LustreError: 71804:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2396.400585] Lustre: server umount lustre-MDT0000 complete [ 2396.404717] LustreError: 71804:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2396.459382] LustreError: 70772:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2396.477506] LustreError: 70772:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2403.262094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2403.408381] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2403.796986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2403.868285] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2407.737311] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2408.984886] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2409.031128] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2409.112539] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2409.289404] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2419.687051] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2419.702055] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2419.718309] Lustre: Skipped 1 previous similar message [ 2419.722947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2422.250603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2422.250712] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2425.646694] Lustre: server umount lustre-OST0000 complete [ 2432.494364] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2432.505682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2432.512443] Lustre: Skipped 1 previous similar message [ 2432.543078] Lustre: Skipped 2 previous similar messages [ 2435.773683] Lustre: server umount lustre-MDT0000 complete [ 2437.602107] LustreError: 68726:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2437.628546] LustreError: 68726:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 2439.205836] LustreError: 68711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787117528 with bad export cookie 4631761737729795605 [ 2439.213969] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2439.223993] LustreError: 68711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2439.608818] Lustre: server umount lustre-MDT0001 complete [ 2451.148516] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 2454.116137] Key type lgssc unregistered [ 2454.366538] LNet: 73595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2454.373125] LNetError: 73595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2454.393041] LNet: Removed LNI 192.168.201.126@tcp [ 2455.329866] Key type .llcrypt unregistered [ 2455.334155] Key type ._llcrypt unregistered [ 2478.867950] Key type ._llcrypt registered [ 2478.870142] Key type .llcrypt registered [ 2478.950768] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 2480.081997] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2480.172106] alg: No test for adler32 (adler32-zlib) [ 2481.377160] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 2481.653756] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 2483.303174] Key type lgssc registered [ 2484.266574] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2492.513538] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2492.556164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2494.160862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2498.780873] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2504.672503] LustreError: 75007:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2504.691552] LustreError: 75007:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2508.170702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2508.624636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2513.947047] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2522.610760] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2527.901718] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2534.657268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2535.171355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2539.252394] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2540.889761] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2548.383183] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2558.224724] Lustre: Failing over lustre-MDT0000 [ 2558.626427] Lustre: server umount lustre-MDT0000 complete [ 2559.973901] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2559.983817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2559.988123] LustreError: 75002:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2559.995717] Lustre: Skipped 1 previous similar message [ 2560.020048] LustreError: 75002:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2561.385286] LustreError: 75002:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2565.110256] LustreError: 75007:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2565.128806] LustreError: 75007:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2567.839059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2567.972668] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2568.167603] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2568.361214] LustreError: 78098:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2568.368150] LustreError: 78098:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.126@tcp: cfg command failed: rc = -9 [ 2568.383200] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2568.383200] [ 2568.398335] LustreError: MGC192.168.201.126@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2568.422370] LustreError: 78086:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2568.436183] LustreError: 78086:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2568.447098] LustreError: 78086:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2568.451281] LustreError: 78086:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2568.473908] Lustre: server umount lustre-MDT0000 complete [ 2568.477850] LustreError: 78086:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2570.211416] LustreError: 75004:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2570.229777] LustreError: 75004:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2576.611500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2576.910649] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2576.927487] LustreError: 74999:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2577.391196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2577.515747] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2577.789540] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2582.535914] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2582.566715] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2582.617845] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2583.186618] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2592.741718] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2592.763858] Lustre: Skipped 1 previous similar message [ 2592.770193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2592.776319] Lustre: Skipped 1 previous similar message [ 2597.861514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2597.866986] Lustre: Skipped 1 previous similar message [ 2598.898655] Lustre: server umount lustre-OST0000 complete [ 2602.976026] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2602.977287] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2602.978936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2602.983723] Lustre: Skipped 2 previous similar messages [ 2608.096378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2608.110748] Lustre: Skipped 2 previous similar messages [ 2608.694061] Lustre: server umount lustre-MDT0000 complete [ 2612.636082] LustreError: 74988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787117701 with bad export cookie 16832720015134966531 [ 2612.653562] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2613.294420] Lustre: server umount lustre-MDT0001 complete [ 2624.927125] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 2627.815526] Key type lgssc unregistered [ 2628.078536] LNet: 79882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2628.083270] LNetError: 79882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2628.099824] LNet: Removed LNI 192.168.201.126@tcp [ 2629.264402] Key type .llcrypt unregistered [ 2629.270297] Key type ._llcrypt unregistered [ 2644.995864] Key type ._llcrypt registered [ 2644.998852] Key type .llcrypt registered [ 2645.117843] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 2645.506357] Key type .llcrypt unregistered [ 2645.508046] Key type ._llcrypt unregistered [ 2675.864216] Key type ._llcrypt registered [ 2675.866406] Key type .llcrypt registered [ 2675.992468] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 2677.945307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2677.977159] alg: No test for adler32 (adler32-zlib) [ 2679.093920] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 2679.376301] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 2681.087314] Key type lgssc registered [ 2682.683325] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2690.809721] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2690.830461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2692.419396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2698.250964] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2702.835782] LustreError: 81787:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2702.861432] LustreError: 81787:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2707.962873] LustreError: 81788:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2708.262115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2708.447228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2713.586538] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2721.714507] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2728.051796] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2735.636793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2735.997058] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2738.027393] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2743.097780] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2750.106645] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2763.744457] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2763.744708] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2763.754322] LustreError: Skipped 1 previous similar message [ 2763.757031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2763.776216] Lustre: Skipped 1 previous similar message [ 2768.867616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2768.873892] Lustre: Skipped 1 previous similar message [ 2769.296937] Lustre: server umount lustre-OST0000 complete [ 2773.985668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2773.999286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2774.008282] Lustre: Skipped 1 previous similar message [ 2779.105364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2779.121598] Lustre: Skipped 2 previous similar messages [ 2779.481866] Lustre: server umount lustre-MDT0000 complete [ 2783.108771] LustreError: 81767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787117872 with bad export cookie 529933698218026718 [ 2783.122080] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2783.131505] LustreError: 81767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2783.447662] Lustre: server umount lustre-MDT0001 complete [ 2793.217675] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 2795.930974] Key type lgssc unregistered [ 2796.160361] LNet: 85341:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2796.168835] LNetError: 85341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2796.185467] LNet: Removed LNI 192.168.201.126@tcp [ 2796.852279] Key type .llcrypt unregistered [ 2796.856172] Key type ._llcrypt unregistered [ 2816.741574] Key type ._llcrypt registered [ 2816.746413] Key type .llcrypt registered [ 2816.922349] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 2818.214873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2818.301344] alg: No test for adler32 (adler32-zlib) [ 2819.508297] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 2819.827294] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 2821.599403] Key type lgssc registered [ 2822.792337] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2831.264917] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2831.287782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2832.884976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2838.460324] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2842.086674] LustreError: 86732:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2842.108955] LustreError: 86732:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2847.211480] LustreError: 86733:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2847.998569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2848.428158] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2853.661403] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2861.036781] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2867.608345] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2874.252962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2874.840259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2877.954047] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2881.917399] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2889.818974] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2903.528195] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2903.535090] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2903.543655] LustreError: Skipped 1 previous similar message [ 2903.564148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2903.570773] Lustre: Skipped 1 previous similar message [ 2905.961174] Lustre: server umount lustre-OST0000 complete [ 2913.762478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2913.777497] Lustre: Skipped 1 previous similar message [ 2913.791678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2913.801963] Lustre: Skipped 1 previous similar message [ 2916.386913] Lustre: server umount lustre-MDT0000 complete [ 2918.886480] LustreError: 86732:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2918.914956] LustreError: 86732:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2919.687863] LustreError: 86715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787118008 with bad export cookie 10548743324120407605 [ 2919.691989] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2919.704819] LustreError: 86715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2920.103395] Lustre: server umount lustre-MDT0001 complete [ 2928.832412] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 2930.968576] Key type lgssc unregistered [ 2931.195791] LNet: 90268:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2931.202873] LNetError: 90268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2931.217443] LNet: Removed LNI 192.168.201.126@tcp [ 2931.803176] Key type .llcrypt unregistered [ 2931.805672] Key type ._llcrypt unregistered [ 2950.704552] Key type ._llcrypt registered [ 2950.708025] Key type .llcrypt registered [ 2950.803721] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 2951.895137] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2951.942333] alg: No test for adler32 (adler32-zlib) [ 2952.978874] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 2953.194416] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 2954.823202] Key type lgssc registered [ 2955.847067] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2963.757587] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2963.790427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2965.343370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2970.736135] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2975.718381] LustreError: 91661:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2975.736167] LustreError: 91661:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2978.560620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2978.808551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2984.342602] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2991.360930] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2997.416044] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3005.946197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3006.431332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3010.555789] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3014.982357] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3024.502439] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3036.130456] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3036.135161] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3036.139192] LustreError: Skipped 1 previous similar message [ 3036.222100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3040.244230] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3040.257704] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3040.258261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3041.919092] Lustre: server umount lustre-OST0000 complete [ 3050.472814] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3050.474124] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3050.486565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3050.503506] Lustre: Skipped 1 previous similar message [ 3050.504632] Lustre: Skipped 1 previous similar message [ 3055.584846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3055.594039] Lustre: Skipped 2 previous similar messages [ 3060.191332] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3060.362949] Lustre: server umount lustre-MDT0000 complete [ 3060.725167] LustreError: 92388:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3060.744525] LustreError: 92388:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3064.086540] LustreError: 91641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787118153 with bad export cookie 15331457396326596458 [ 3064.104311] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3064.110192] LustreError: 91641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3064.419923] Lustre: server umount lustre-MDT0001 complete [ 3076.978379] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 3079.855861] Key type lgssc unregistered [ 3080.122628] LNet: 95196:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3080.135935] LNetError: 95196:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3081.189188] LNet: Removed LNI 192.168.201.126@tcp [ 3082.151365] Key type .llcrypt unregistered [ 3082.153968] Key type ._llcrypt unregistered [ 3107.349717] Key type ._llcrypt registered [ 3107.352805] Key type .llcrypt registered [ 3107.467065] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 3108.943610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3108.984522] alg: No test for adler32 (adler32-zlib) [ 3110.163751] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 3110.469113] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 3112.159183] Key type lgssc registered [ 3113.412691] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3121.854384] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3121.880405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3137.439538] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3142.560975] LustreError: 96600:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.126@tcp: failed processing log, type 4: rc = -110 [ 3169.311251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3175.634742] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3183.690868] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3190.281862] Lustre: server umount lustre-OST0000 complete [ 3200.632808] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 01:44:47 (1787118287) [ 3208.395783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3208.943606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3213.583651] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3214.315242] LustreError: 98240:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3214.338347] LustreError: 98240:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3218.408339] LustreError: 98241:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3221.647232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3222.178417] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3226.392190] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3234.182516] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3240.310266] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3247.412657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3247.753520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3253.863343] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3254.661038] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3262.152366] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3277.796451] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3277.800540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3277.812400] Lustre: Skipped 1 previous similar message [ 3277.840886] Lustre: Skipped 1 previous similar message [ 3282.922246] Lustre: server umount lustre-OST0000 complete [ 3288.052168] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3288.058049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3288.085937] Lustre: Skipped 1 previous similar message [ 3288.102473] Lustre: Skipped 1 previous similar message [ 3293.159373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3293.165269] Lustre: Skipped 1 previous similar message [ 3293.615514] Lustre: server umount lustre-MDT0000 complete [ 3297.252014] LustreError: 98969:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3297.275822] LustreError: 98969:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3297.651420] LustreError: 98222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787118386 with bad export cookie 6872811521579552528 [ 3297.656600] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3297.670774] LustreError: 98222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3297.964235] Lustre: server umount lustre-MDT0001 complete [ 3308.555373] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 3311.346471] Key type lgssc unregistered [ 3311.686573] LNet: 101933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3311.709782] LNetError: 101933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3311.731732] LNet: Removed LNI 192.168.201.126@tcp [ 3312.835458] Key type .llcrypt unregistered [ 3312.843714] Key type ._llcrypt unregistered [ 3355.989720] Key type ._llcrypt registered [ 3355.992701] Key type .llcrypt registered [ 3356.098965] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 3357.326121] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3357.455711] alg: No test for adler32 (adler32-zlib) [ 3358.614294] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 3359.019712] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 3360.687210] Key type lgssc registered [ 3362.306358] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3370.697139] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3370.738776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3372.094661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3376.993054] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3382.243552] LustreError: 104118:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3382.278305] LustreError: 104118:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3386.560751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3386.834620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3392.593243] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3401.842410] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3408.229284] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3415.591042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3416.027816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3417.096831] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3423.325883] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3431.541795] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3441.632863] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3441.638241] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3441.661130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3441.697099] Lustre: Skipped 1 previous similar message [ 3446.757459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3446.770519] Lustre: Skipped 1 previous similar message [ 3447.289583] Lustre: server umount lustre-OST0000 complete [ 3451.872530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3451.890615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3451.894896] Lustre: Skipped 1 previous similar message [ 3451.915638] Lustre: Skipped 2 previous similar messages [ 3456.993861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3457.004542] Lustre: Skipped 1 previous similar message [ 3458.028541] Lustre: server umount lustre-MDT0000 complete [ 3462.115165] LustreError: 104118:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3462.132679] LustreError: 104118:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3462.415236] LustreError: 104098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787118551 with bad export cookie 11800279425286571936 [ 3462.446387] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3463.135323] Lustre: server umount lustre-MDT0001 complete [ 3476.336078] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 3479.538079] Key type lgssc unregistered [ 3479.865503] LNet: 107652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3479.875556] LNetError: 107652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3479.895235] LNet: Removed LNI 192.168.201.126@tcp [ 3480.873582] Key type .llcrypt unregistered [ 3480.875857] Key type ._llcrypt unregistered [ 3498.867275] Key type ._llcrypt registered [ 3498.870667] Key type .llcrypt registered [ 3498.990896] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 3500.402490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3500.435682] alg: No test for adler32 (adler32-zlib) [ 3501.533696] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 3501.761056] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 3503.552759] Key type lgssc registered [ 3505.346094] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3514.451712] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3514.498383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3530.015717] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3535.135344] LustreError: 108898:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.126@tcp: failed processing log, type 4: rc = -110 [ 3561.823192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3568.976114] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3577.053781] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3584.502783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3584.756046] LustreError: MGC192.168.201.126@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3584.765806] LustreError: 109924:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3584.771679] LustreError: 109924:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3584.780208] LustreError: 109924:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3584.790072] LustreError: 108906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787118673 with bad export cookie 18077643101616100474 [ 3584.809879] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3584.878387] Lustre: server umount lustre-MDT0000 complete [ 3584.882307] LustreError: 109924:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3613.290250] Lustre: server umount lustre-OST0000 complete [ 3623.220067] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 3636.050230] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 3677.418357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3677.660956] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3677.703173] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3677.799817] Lustre: lustre-MDT0000: new disk, initializing [ 3677.937668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3677.984398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3683.731716] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3694.863245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3694.961107] Lustre: 112718:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3695.007272] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3695.011375] Lustre: Skipped 1 previous similar message [ 3695.070188] Lustre: lustre-MDT0001: new disk, initializing [ 3695.121441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3695.157869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3695.170762] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3699.971688] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3709.077936] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3714.405328] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3720.943903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3721.389329] Lustre: lustre-OST0000: new disk, initializing [ 3721.397770] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3721.407466] Lustre: 114686:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3721.521727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3728.250403] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3730.035205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3730.058937] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3730.228694] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3738.107584] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3743.653513] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3743.874606] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3748.553727] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3748.734318] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3750.882055] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3750.885192] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3750.890827] LustreError: Skipped 1 previous similar message [ 3750.901042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3750.918973] Lustre: Skipped 1 previous similar message [ 3750.936918] Lustre: Skipped 1 previous similar message [ 3756.005797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3756.015748] Lustre: Skipped 1 previous similar message [ 3757.109461] Lustre: server umount lustre-OST0000 complete [ 3761.634636] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3761.639889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3761.645734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3766.243112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3766.246333] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3767.884230] Lustre: server umount lustre-MDT0000 complete [ 3771.631861] LustreError: 112710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787118860 with bad export cookie 18077643101616100579 [ 3771.635819] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3771.647495] LustreError: 112710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3771.928714] Lustre: server umount lustre-MDT0001 complete [ 3780.368859] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 01:54:27 (1787118867) [ 3783.683603] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3800.310294] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 3809.409800] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 3844.667991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3845.077990] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3845.123774] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3845.253327] Lustre: lustre-MDT0000: new disk, initializing [ 3845.418900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3845.440856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3851.065620] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3865.454598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3865.705909] Lustre: 119602:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3865.820440] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3865.824851] Lustre: Skipped 1 previous similar message [ 3865.988255] Lustre: lustre-MDT0001: new disk, initializing [ 3866.122702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3866.182582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3866.192449] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3872.372956] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3882.026718] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3888.484359] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3895.810785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3896.054846] Lustre: lustre-OST0000: new disk, initializing [ 3896.060438] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3896.066326] Lustre: 121573:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3896.174741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3902.087666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3902.114326] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3902.272387] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3904.078275] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3913.591814] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3918.787728] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3919.061406] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3923.425635] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3923.622909] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3928.039317] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3928.049470] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3928.066877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3928.069547] Lustre: Skipped 1 previous similar message [ 3931.776679] Lustre: server umount lustre-OST0000 complete [ 3937.761147] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3937.767652] LustreError: Skipped 1 previous similar message [ 3937.770360] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3937.784290] Lustre: Skipped 1 previous similar message [ 3937.787964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3941.697937] Lustre: server umount lustre-MDT0000 complete [ 3943.403677] LustreError: 119616:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3943.427506] LustreError: 119616:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3945.848504] LustreError: 119595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787119034 with bad export cookie 18077643101616101461 [ 3945.852671] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3945.858179] LustreError: 119595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3946.235821] Lustre: server umount lustre-MDT0001 complete [ 3954.680161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3955.247040] LustreError: 123788:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3955.340449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3960.081645] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3960.804675] LustreError: 123789:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3965.921322] LustreError: 123788:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3968.168508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3973.105501] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3981.699553] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3987.431098] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3995.574836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4002.756222] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4010.529253] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4020.045747] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4024.806460] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4024.837593] Lustre: Skipped 1 previous similar message [ 4024.843756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4024.849902] Lustre: Skipped 2 previous similar messages [ 4024.854373] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4029.932348] LustreError: 125612:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4029.961327] LustreError: 125612:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4030.120843] Lustre: server umount lustre-OST0000 complete [ 4035.048594] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4039.984304] Lustre: server umount lustre-MDT0000 complete [ 4040.168707] LustreError: 124517:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4043.468995] LustreError: 123770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787119132 with bad export cookie 18077643101616102343 [ 4043.477617] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4043.481213] LustreError: 123770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4043.974152] Lustre: server umount lustre-MDT0001 complete [ 4058.437692] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 4062.845270] Key type lgssc unregistered [ 4063.151519] LNet: 127418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4063.161055] LNetError: 127418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4063.176496] LNet: Removed LNI 192.168.201.126@tcp [ 4064.229712] Key type .llcrypt unregistered [ 4064.233364] Key type ._llcrypt unregistered [ 4083.173648] Key type ._llcrypt registered [ 4083.175473] Key type .llcrypt registered [ 4083.287824] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 4097.448872] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 4098.673236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4098.724898] alg: No test for adler32 (adler32-zlib) [ 4099.854505] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 4100.088797] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 4101.791366] Key type lgssc registered [ 4103.457111] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4147.327761] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4147.388048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4148.877629] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4148.953324] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4149.197616] Lustre: lustre-MDT0000: new disk, initializing [ 4149.363535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4149.397354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4155.219448] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4166.527561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4166.586317] Lustre: 130573:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4166.628489] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4166.634229] Lustre: Skipped 1 previous similar message [ 4166.694536] Lustre: lustre-MDT0001: new disk, initializing [ 4166.827899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4166.850777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4166.857585] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4171.785619] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4182.685653] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4189.319707] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4195.798469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4196.165186] Lustre: lustre-OST0000: new disk, initializing [ 4196.169221] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4196.184888] Lustre: 132555:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4196.296310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4202.477369] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4204.072618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4204.086791] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4204.227786] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4212.116868] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4216.218741] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4216.468084] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4221.026297] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4221.257984] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4224.997481] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4224.999629] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4225.010359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4225.030822] Lustre: Skipped 1 previous similar message [ 4229.452290] Lustre: server umount lustre-OST0000 complete [ 4235.237483] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4235.247520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4235.264384] Lustre: Skipped 1 previous similar message [ 4235.289505] Lustre: Skipped 2 previous similar messages [ 4240.353139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4240.358995] Lustre: Skipped 1 previous similar message [ 4240.717795] Lustre: server umount lustre-MDT0000 complete [ 4244.686289] LustreError: 130564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787119333 with bad export cookie 16537029417568794719 [ 4244.701405] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4244.703197] LustreError: 130564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4245.282885] Lustre: server umount lustre-MDT0001 complete [ 4255.018786] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 02:02:22 (1787119342) [ 4262.340488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4262.977470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4267.963922] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4268.006303] LustreError: 134968:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4268.035314] LustreError: 134968:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4272.102565] LustreError: 134969:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4277.222904] LustreError: 134968:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4277.462847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4278.033417] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4283.251578] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4290.530437] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4296.769613] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4302.816159] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4302.844657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4302.858387] Lustre: Skipped 1 previous similar message [ 4305.542709] Lustre: server umount lustre-MDT0000 complete [ 4308.960865] LustreError: 134965:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4309.000137] LustreError: 134965:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4309.580240] LustreError: 134949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787119398 with bad export cookie 16537029417568795601 [ 4309.591655] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4310.023587] Lustre: server umount lustre-MDT0001 complete [ 4319.013868] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 02:03:26 (1787119406) [ 4325.540620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4341.090537] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4346.207456] LustreError: 137384:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.126@tcp: failed processing log, type 4: rc = -110 [ 4371.871612] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4380.816678] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4390.359747] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4392.835298] Lustre: server umount lustre-OST0000 complete [ 4401.172394] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 02:04:48 (1787119488) [ 4407.352038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4407.919747] LustreError: 138830:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4407.998665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4412.734238] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4417.505951] LustreError: 138830:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4417.531829] LustreError: 138830:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4420.888188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4425.509436] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4432.407630] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4437.959534] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4446.130563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4446.423189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4446.426781] Lustre: Skipped 1 previous similar message [ 4452.881848] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4461.068479] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4477.411359] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4477.412776] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4477.424468] LustreError: Skipped 1 previous similar message [ 4477.441184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4477.450301] Lustre: Skipped 3 previous similar messages [ 4477.479594] Lustre: Skipped 4 previous similar messages [ 4480.993873] Lustre: server umount lustre-MDT0000 complete [ 4482.535495] LustreError: 140954:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4482.554091] LustreError: 140954:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4484.545049] LustreError: 140673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787119573 with bad export cookie 16537029417568796266 [ 4484.548321] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4484.554381] LustreError: 140673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4484.938134] Lustre: server umount lustre-MDT0001 complete [ 4499.149794] Lustre: server umount lustre-OST0000 complete [ 4509.280975] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 02:06:36 (1787119596) [ 4515.780158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4516.128429] LustreError: 142506:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4516.199595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4522.491294] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4531.579504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4537.020161] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4544.649933] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4550.990455] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4557.262390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4561.800383] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4563.979183] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4572.830364] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4578.706503] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4579.017710] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4585.025373] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4585.213331] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4587.491245] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4587.502681] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4587.521995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4591.612030] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4593.398238] Lustre: server umount lustre-OST0000 complete [ 4598.756861] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4598.763063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4598.780360] Lustre: Skipped 1 previous similar message [ 4603.430310] Lustre: server umount lustre-MDT0000 complete [ 4606.946045] LustreError: 142507:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4606.971611] LustreError: 142507:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4607.125965] LustreError: 142488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787119696 with bad export cookie 16537029417568797337 [ 4607.132533] LustreError: 142488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4607.135076] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4607.446214] Lustre: server umount lustre-MDT0001 complete [ 4614.215697] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 02:08:22 (1787119702) [ 4620.514682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4636.192370] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4641.316136] LustreError: 146665:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.126@tcp: failed processing log, type 4: rc = -110 [ 4667.040560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4667.043030] Lustre: Skipped 2 previous similar messages [ 4674.424413] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4682.723568] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4689.985497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4690.639325] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4695.959487] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4705.058639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4710.030295] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4717.602178] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4723.019610] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4727.679851] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4727.937368] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4733.089958] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4733.286612] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4735.970131] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4735.975606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4735.985219] Lustre: Skipped 1 previous similar message [ 4735.989621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4735.995618] Lustre: Skipped 5 previous similar messages [ 4741.566070] Lustre: server umount lustre-MDT0000 complete [ 4741.604579] LustreError: 147701:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4741.613166] LustreError: 147701:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 4745.494790] LustreError: 146673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787119834 with bad export cookie 16537029417568798359 [ 4745.501272] LustreError: 146673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4745.501413] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4745.896322] Lustre: server umount lustre-MDT0001 complete [ 4760.677625] Lustre: server umount lustre-OST0000 complete [ 4771.132735] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 02:10:58 (1787119858) [ 4779.102638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4794.784302] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4799.905205] LustreError: 150859:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.126@tcp: failed processing log, type 4: rc = -110 [ 4833.430501] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4840.669362] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4847.443764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4847.901457] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4852.209589] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4860.235629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4864.765407] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4871.529134] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4877.672409] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4885.762290] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4886.063285] Lustre: lustre-OST0001: new disk, initializing [ 4886.078515] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4886.092875] Lustre: 153736:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4891.962844] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4895.859299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4895.880138] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4896.022806] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4901.841768] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4906.553959] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4906.721531] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4911.976467] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4912.286749] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4914.663905] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4914.674283] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4914.691988] Lustre: Skipped 2 previous similar messages [ 4914.698862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4914.705115] Lustre: Skipped 3 previous similar messages [ 4920.069564] Lustre: server umount lustre-OST0000 complete [ 4931.066607] Lustre: server umount lustre-OST0001 complete [ 4941.553764] Lustre: server umount lustre-MDT0000 complete [ 4945.531141] LustreError: 150865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120034 with bad export cookie 16537029417568799311 [ 4945.556912] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4946.473394] Lustre: server umount lustre-MDT0001 complete [ 4974.038330] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 02:14:21 (1787120061) [ 4975.949747] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4977.946207] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4979.813894] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 02:14:27 (1787120067) [ 4986.897033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4987.171135] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4987.196801] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4987.397580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4987.401898] Lustre: Skipped 6 previous similar messages [ 4992.495847] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5002.272568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5002.441963] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5002.480129] Lustre: 157296:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5007.525891] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5014.703071] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5020.372990] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5027.288219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5027.448053] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5033.906790] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5036.209617] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5042.706332] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5048.819221] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5049.086789] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5055.592180] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5055.875432] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5062.115494] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5062.116452] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5062.117722] LustreError: Skipped 2 previous similar messages [ 5062.121360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5062.121369] Lustre: Skipped 6 previous similar messages [ 5062.123659] Lustre: Skipped 6 previous similar messages [ 5064.480435] Lustre: server umount lustre-OST0000 complete [ 5085.751942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5092.326524] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5100.855363] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5102.405378] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5102.429562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5102.444447] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5106.321687] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5106.541208] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5111.343513] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5111.604383] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5117.503534] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9c1944abd000.ost_server_uuid 50 [ 5119.116975] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9c1944abd000.ost_server_uuid in FULL state after 0 sec [ 5119.320389] LustreError: 157302:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.201.26@tcp) returned error from blocking AST (req@000000008236ae06 x1873930290988032 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000078e89ebc/0xe57f54a36a5f93a9 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.26@tcp remote: 0xe0a10bccd285c2e9 expref: 5 pid: 157303 timeout: 5219 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5119.343188] LustreError: lustre-MDT0000: A client on nid 192.168.201.26@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5119.348780] LustreError: 157292:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.26@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000078e89ebc/0xe57f54a36a5f93a9 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.26@tcp remote: 0xe0a10bccd285c2e9 expref: 6 pid: 157303 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5128.167898] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5128.179092] LustreError: Skipped 1 previous similar message [ 5130.639201] Lustre: server umount lustre-OST0000 complete [ 5140.986335] Lustre: server umount lustre-MDT0000 complete [ 5143.521020] LustreError: 157304:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5143.560806] LustreError: 157304:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 5145.374081] LustreError: 157288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120234 with bad export cookie 16537029417568800494 [ 5145.376161] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5145.387216] LustreError: 157288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5151.952941] Lustre: server umount lustre-MDT0001 complete [ 5163.055372] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 5165.575383] Key type lgssc unregistered [ 5165.841929] LNet: 163407:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5165.847970] LNetError: 163407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5165.860633] LNet: Removed LNI 192.168.201.126@tcp [ 5166.883317] Key type .llcrypt unregistered [ 5166.890876] Key type ._llcrypt unregistered [ 5190.207581] Key type ._llcrypt registered [ 5190.216888] Key type .llcrypt registered [ 5190.355454] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 5191.453371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5191.518296] alg: No test for adler32 (adler32-zlib) [ 5192.696255] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 5193.004603] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 5194.743190] Key type lgssc registered [ 5196.020458] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5203.611184] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5203.641389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5205.118937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5209.692691] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5215.717143] LustreError: 164825:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5215.730435] LustreError: 164825:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5218.384769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5218.742918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5223.177526] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5230.654490] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5237.374934] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5244.894369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5245.280413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5250.424278] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5252.564295] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5261.079251] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5271.091371] Lustre: Failing over lustre-MDT0000 [ 5271.480166] Lustre: server umount lustre-MDT0000 complete [ 5275.105874] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5275.128764] LustreError: 164820:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5275.143281] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5275.156414] LustreError: 164820:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5280.250156] LustreError: 164826:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5280.272070] LustreError: 164826:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5284.105059] LustreError: 166669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.26@tcp arrived at 1787120373 with bad export cookie 13665903457655707989 [ 5284.120871] LustreError: 166669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5285.349177] LustreError: 164825:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5285.379499] LustreError: 164825:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5290.468410] LustreError: 165554:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5290.494390] LustreError: 165554:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5291.495128] Lustre: 164134:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787120364/real 1787120364] req@000000000d3c46e4 x1873931435797888/t0(0) o400->MGC192.168.201.126@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787120380 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5291.529823] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5291.745373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5301.729232] LustreError: 164820:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5301.747685] LustreError: 164820:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 5301.747985] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xbda70aca2859d9be [ 5301.775840] Lustre: MGC192.168.201.126@tcp: Connection restored to 0@lo (at 0@lo) [ 5302.000461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5302.057197] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5306.226598] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5307.367230] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5307.381875] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5308.498956] Lustre: lustre-MDT0000: Denying connection for new client f99c03e5-9bfd-4258-8d22-4a759cd910c6 (at 192.168.201.26@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:09 [ 5317.611333] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5317.617375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5317.632117] Lustre: Skipped 3 previous similar messages [ 5317.641185] Lustre: Skipped 1 previous similar message [ 5322.725986] LustreError: 166645:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5322.754167] LustreError: 166645:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5322.895608] Lustre: server umount lustre-OST0000 complete [ 5327.428427] LustreError: 168558:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5327.439743] Lustre: 167907:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5327.458640] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5327.471973] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5327.484664] LustreError: 167907:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000940d56f6 x1873931435817600/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5327.496053] LustreError: 167907:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5327.507350] LustreError: 167907:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5327.514950] Lustre: lustre-MDT0000: Recovery over after 0:20, of 2 clients 0 recovered and 2 were evicted. [ 5327.841786] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5327.899600] Lustre: server umount lustre-MDT0000 complete [ 5331.996176] LustreError: 166669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120420 with bad export cookie 13665903457655708094 [ 5332.007068] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5332.008871] LustreError: 166669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5334.449176] Lustre: server umount lustre-MDT0001 complete [ 5345.989704] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 5348.799513] Key type lgssc unregistered [ 5349.037886] LNet: 169311:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5349.053445] LNetError: 169311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5349.078625] LNet: Removed LNI 192.168.201.126@tcp [ 5350.345401] Key type .llcrypt unregistered [ 5350.347179] Key type ._llcrypt unregistered [ 5374.130677] Key type ._llcrypt registered [ 5374.133714] Key type .llcrypt registered [ 5374.276701] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 5375.397127] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5375.521683] alg: No test for adler32 (adler32-zlib) [ 5376.695524] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 5376.957751] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 5378.711412] Key type lgssc registered [ 5380.231250] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5388.083739] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5388.097183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5389.729259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5394.153720] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5399.017552] LustreError: 170724:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5399.046850] LustreError: 170724:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5402.698970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5403.135105] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5407.148878] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5414.664318] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5420.849938] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5427.947405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5428.365737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5434.486482] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5434.546400] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5441.672744] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5450.213528] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5450.218405] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5450.223516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5450.238988] Lustre: Skipped 1 previous similar message [ 5450.240150] LustreError: Skipped 1 previous similar message [ 5453.114556] Lustre: server umount lustre-OST0000 complete [ 5459.434762] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5459.446491] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5459.463597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5459.467985] Lustre: Skipped 1 previous similar message [ 5460.451279] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5460.482585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5460.488797] Lustre: Skipped 1 previous similar message [ 5465.570362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5465.578692] Lustre: Skipped 1 previous similar message [ 5470.695988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5470.710340] Lustre: Skipped 1 previous similar message [ 5472.223134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5472.509609] Lustre: server umount lustre-MDT0000 complete [ 5475.808937] LustreError: 172564:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5475.841240] LustreError: 172564:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5476.941592] LustreError: 170705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120565 with bad export cookie 15472507394670801736 [ 5476.973052] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5476.980510] LustreError: 170705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5477.479629] Lustre: server umount lustre-MDT0001 complete [ 5489.802828] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 5493.699333] Key type lgssc unregistered [ 5494.110077] LNet: 174260:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5494.126887] LNetError: 174260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5494.218414] LNet: Removed LNI 192.168.201.126@tcp [ 5495.191082] Key type .llcrypt unregistered [ 5495.192900] Key type ._llcrypt unregistered [ 5529.379601] Key type ._llcrypt registered [ 5529.381790] Key type .llcrypt registered [ 5529.529583] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 5531.515749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5531.541869] alg: No test for adler32 (adler32-zlib) [ 5533.003778] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 5533.461822] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 5535.143172] Key type lgssc registered [ 5537.054178] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5546.991175] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5547.012048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5548.413608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5552.924574] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5558.756463] LustreError: 175964:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5558.789341] LustreError: 175964:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5563.577111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5563.893336] LustreError: 175965:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5564.291183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5570.829362] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5580.320463] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5586.323837] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5593.550401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5594.063219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5599.232330] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5600.721479] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5608.039293] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5619.695062] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5619.704518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5619.710625] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5619.735429] Lustre: Skipped 1 previous similar message [ 5619.770633] Lustre: Skipped 1 previous similar message [ 5622.658284] Lustre: server umount lustre-OST0000 complete [ 5629.929726] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5629.959529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5635.054672] LustreError: 175965:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5635.151920] Lustre: server umount lustre-MDT0000 complete [ 5639.147927] LustreError: 175965:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5639.169381] LustreError: 175965:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5639.377234] LustreError: 175945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120728 with bad export cookie 10286683585792246006 [ 5639.380736] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5639.402242] LustreError: 175945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5640.220721] Lustre: server umount lustre-MDT0001 complete [ 5652.792365] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 5656.300948] Key type lgssc unregistered [ 5656.721709] LNet: 179499:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5656.733610] LNetError: 179499:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5656.762544] LNet: Removed LNI 192.168.201.126@tcp [ 5658.124189] Key type .llcrypt unregistered [ 5658.129934] Key type ._llcrypt unregistered [ 5682.458725] Key type ._llcrypt registered [ 5682.460756] Key type .llcrypt registered [ 5682.556383] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 5683.750407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5683.921354] alg: No test for adler32 (adler32-zlib) [ 5684.908258] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 5685.001137] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 5686.623185] Key type lgssc registered [ 5687.879888] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5700.260645] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5700.297510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5701.600857] Lustre: *** cfs_fail_loc=135, val=0*** [ 5701.677833] LustreError: 181023:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5701.685348] LustreError: 181023:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.126@tcp: cfg command failed: rc = -2 [ 5701.691377] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5701.691377] [ 5701.701846] LustreError: MGC192.168.201.126@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5701.718974] LustreError: 180984:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5701.723358] LustreError: 180984:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5701.737704] LustreError: 180984:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5701.741955] LustreError: 180984:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5701.823232] Lustre: server umount lustre-MDT0000 complete [ 5701.825734] LustreError: 180984:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5708.224435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5708.751334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5713.107831] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5719.027242] LustreError: 181395:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5719.048456] LustreError: 181395:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5722.926103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5723.390881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5727.368386] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5734.333514] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5739.869250] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5743.014593] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5743.019171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5743.034502] Lustre: Skipped 1 previous similar message [ 5743.050091] Lustre: Skipped 1 previous similar message [ 5748.066888] Lustre: server umount lustre-MDT0000 complete [ 5748.200306] LustreError: 181394:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5748.217984] LustreError: 181394:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5751.831709] LustreError: 181375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120840 with bad export cookie 714149049712923865 [ 5751.841955] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5751.856420] LustreError: 181375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5752.175927] Lustre: server umount lustre-MDT0001 complete [ 5762.460102] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 5765.346262] Key type lgssc unregistered [ 5765.738311] LNet: 183714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5765.756718] LNetError: 183714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5766.831985] LNet: Removed LNI 192.168.201.126@tcp [ 5767.853125] Key type .llcrypt unregistered [ 5767.855974] Key type ._llcrypt unregistered [ 5791.348401] Key type ._llcrypt registered [ 5791.355335] Key type .llcrypt registered [ 5791.456656] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 5791.773423] Key type .llcrypt unregistered [ 5791.778423] Key type ._llcrypt unregistered [ 5808.742975] Key type ._llcrypt registered [ 5808.750555] Key type .llcrypt registered [ 5808.922991] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 5809.754812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5809.812741] alg: No test for adler32 (adler32-zlib) [ 5810.949816] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 5811.253371] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 5812.992145] Key type lgssc registered [ 5814.148957] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5822.014456] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5822.034811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5837.600741] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5842.719382] LustreError: 185851:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.126@tcp: failed processing log, type 4: rc = -110 [ 5869.343223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5875.439653] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5882.433987] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5888.587577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5889.051738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5889.105909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5893.245806] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5894.113461] LustreError: 185876:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5894.123351] LustreError: 185876:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5899.236964] LustreError: 185877:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5900.630122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5901.074958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5904.722631] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5910.571372] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5916.136561] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5919.443591] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5929.977194] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5929.986938] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5929.995834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5930.009112] Lustre: Skipped 1 previous similar message [ 5930.031655] Lustre: Skipped 1 previous similar message [ 5935.110683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5935.122499] Lustre: Skipped 1 previous similar message [ 5940.197586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5940.210683] Lustre: Skipped 1 previous similar message [ 5944.287135] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5944.445702] Lustre: server umount lustre-OST0000 complete [ 5950.433458] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5950.442230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5950.458384] Lustre: Skipped 1 previous similar message [ 5955.552910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5955.564891] Lustre: Skipped 2 previous similar messages [ 5962.207570] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5962.556803] Lustre: server umount lustre-MDT0000 complete [ 5965.797554] LustreError: 186889:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5965.820992] LustreError: 186889:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5967.039368] LustreError: 185858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121056 with bad export cookie 10175744730988399752 [ 5967.043481] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5967.050867] LustreError: 185858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5967.468603] Lustre: server umount lustre-MDT0001 complete [ 5978.924645] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 5981.579934] Key type lgssc unregistered [ 5981.840724] LNet: 189768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5981.848718] LNetError: 189768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5982.889165] LNet: Removed LNI 192.168.201.126@tcp [ 5983.948856] Key type .llcrypt unregistered [ 5983.950443] Key type ._llcrypt unregistered [ 6006.051931] Key type ._llcrypt registered [ 6006.056064] Key type .llcrypt registered [ 6006.180218] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 6007.151641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6007.189306] alg: No test for adler32 (adler32-zlib) [ 6008.183188] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 6008.360336] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 6009.999161] Key type lgssc registered [ 6010.980895] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6019.038864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6019.062636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6020.531911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6024.494487] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6030.823572] LustreError: 191182:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6030.837197] LustreError: 191182:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6032.432622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6032.848255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6037.163452] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6043.826770] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6049.853698] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6057.237478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6057.454891] LustreError: 193004:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6057.690404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6064.044985] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6065.128931] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6071.113238] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6076.727550] Lustre: Failing over lustre-MDT0000 [ 6077.015721] Lustre: server umount lustre-MDT0000 complete [ 6078.945949] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6078.954994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6078.964075] LustreError: 191182:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6080.480517] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6080.501546] Lustre: Skipped 1 previous similar message [ 6084.397830] LustreError: 191178:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6084.424593] LustreError: 191178:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6089.523736] LustreError: 191177:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6089.551257] LustreError: 191177:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6096.868304] Lustre: 190495:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787121169/real 1787121169] req@00000000e2a3a323 x1873932291434240/t0(0) o400->MGC192.168.201.126@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787121185 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6096.893724] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6096.917293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6099.692674] LustreError: 191178:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6099.708725] LustreError: 191178:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 6106.085573] LustreError: 190491:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000da881a2e x1873932291440256/t0(0) o250->MGC192.168.201.126@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6106.518320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6106.615140] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6111.022486] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6111.753689] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6111.763800] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 6111.799995] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6111.889633] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6117.930045] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6126.536653] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6134.752790] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6134.762252] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6134.790912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6135.405329] Lustre: server umount lustre-OST0000 complete [ 6142.433832] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6142.435064] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6142.440786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6142.456619] Lustre: Skipped 1 previous similar message [ 6147.552909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6147.564542] Lustre: Skipped 2 previous similar messages [ 6149.708869] Lustre: server umount lustre-MDT0000 complete [ 6152.681605] LustreError: 191909:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6152.708455] LustreError: 191909:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 6153.318650] LustreError: 191163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121242 with bad export cookie 1262477463351449467 [ 6153.324777] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6153.331022] LustreError: 191163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6153.756112] Lustre: server umount lustre-MDT0001 complete [ 6164.564513] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 6167.377358] Key type lgssc unregistered [ 6167.641936] LNet: 196074:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6167.663566] LNetError: 196074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6167.678816] LNet: Removed LNI 192.168.201.126@tcp [ 6168.556564] Key type .llcrypt unregistered [ 6168.558893] Key type ._llcrypt unregistered [ 6195.090691] Key type ._llcrypt registered [ 6195.101133] Key type .llcrypt registered [ 6195.215574] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 6196.301545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6196.392590] alg: No test for adler32 (adler32-zlib) [ 6197.435208] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 6197.641122] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 6199.343850] Key type lgssc registered [ 6200.667825] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6208.821658] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6208.839410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6210.312335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6215.406839] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6220.769716] LustreError: 197487:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6220.788325] LustreError: 197487:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6221.797609] LustreError: 197488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6224.246658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6224.658181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6229.148644] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6236.733473] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6242.527870] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6249.281451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6249.446871] LustreError: 199310:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6249.721518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6255.791655] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6256.142807] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6263.504251] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6266.547683] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6271.662425] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6289.650744] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6301.153545] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6301.169588] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6301.195198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6302.177744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6302.181229] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6303.863507] Lustre: server umount lustre-OST0000 complete [ 6311.904453] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6311.915216] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6311.939324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6311.944067] Lustre: Skipped 1 previous similar message [ 6313.976350] Lustre: server umount lustre-MDT0000 complete [ 6317.540839] LustreError: 197488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6317.566398] LustreError: 197488:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6317.779831] LustreError: 197469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121406 with bad export cookie 14112167747746809631 [ 6317.780242] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6317.785291] LustreError: 197469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6318.106398] Lustre: server umount lustre-MDT0001 complete [ 6329.145638] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 6331.390742] Key type lgssc unregistered [ 6331.735610] LNet: 201355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6331.741732] LNetError: 201355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6331.760554] LNet: Removed LNI 192.168.201.126@tcp [ 6332.665143] Key type .llcrypt unregistered [ 6332.668083] Key type ._llcrypt unregistered [ 6349.770780] Key type ._llcrypt registered [ 6349.772840] Key type .llcrypt registered [ 6349.852356] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 6351.214775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6351.294976] alg: No test for adler32 (adler32-zlib) [ 6352.279277] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 6352.559575] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 6354.240055] Key type lgssc registered [ 6355.372720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6362.358127] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6362.396694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6363.953660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6368.530586] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6373.366982] LustreError: 202622:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6373.401650] LustreError: 202622:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6378.188281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6378.465169] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6378.618529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6383.386724] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6393.625900] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6402.088374] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6408.859725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6409.169182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6414.847708] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6423.678615] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6433.808181] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6438.499889] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6449.017821] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6460.770756] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6473.668355] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6484.960098] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6484.962418] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6484.970767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6484.982762] Lustre: Skipped 1 previous similar message [ 6484.991631] Lustre: Skipped 1 previous similar message [ 6490.082834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6490.095169] Lustre: Skipped 1 previous similar message [ 6495.199740] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6495.205290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6495.226827] Lustre: Skipped 1 previous similar message [ 6495.380812] Lustre: server umount lustre-OST0000 complete [ 6500.320750] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6500.344260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6500.349794] Lustre: Skipped 1 previous similar message [ 6505.914668] Lustre: server umount lustre-MDT0000 complete [ 6509.598625] LustreError: 202601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121598 with bad export cookie 8222233464526610903 [ 6509.607655] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6509.608758] LustreError: 202601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6510.263303] Lustre: server umount lustre-MDT0001 complete [ 6522.071108] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 6524.671548] Key type lgssc unregistered [ 6524.933597] LNet: 207623:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6524.945726] LNetError: 207623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6524.972210] LNet: Removed LNI 192.168.201.126@tcp [ 6525.758354] Key type .llcrypt unregistered [ 6525.761962] Key type ._llcrypt unregistered [ 6549.895580] Key type ._llcrypt registered [ 6549.898548] Key type .llcrypt registered [ 6550.038644] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 6551.133030] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6551.200162] alg: No test for adler32 (adler32-zlib) [ 6552.322893] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 6552.631113] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 6554.335246] Key type lgssc registered [ 6555.351215] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6563.291292] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6563.315594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6565.055925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6570.332158] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6575.585964] LustreError: 209041:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6575.614897] LustreError: 209041:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6579.544983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6579.967379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6585.431965] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6593.571747] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6599.036504] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6606.513526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6606.949495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6610.061533] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6614.075695] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6621.883364] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6640.119039] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6640.121390] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6640.147313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6640.154713] Lustre: Skipped 1 previous similar message [ 6640.171988] Lustre: Skipped 1 previous similar message [ 6642.841144] Lustre: server umount lustre-OST0000 complete [ 6650.355459] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6650.364085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6650.382435] Lustre: Skipped 1 previous similar message [ 6650.411508] Lustre: Skipped 1 previous similar message [ 6653.555950] Lustre: server umount lustre-MDT0000 complete [ 6655.465910] LustreError: 209040:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6655.493982] LustreError: 209040:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6657.501048] LustreError: 209021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121746 with bad export cookie 18041993222437286604 [ 6657.510032] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6657.519504] LustreError: 209021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6657.806610] Lustre: server umount lustre-MDT0001 complete [ 6670.274295] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 6673.524526] Key type lgssc unregistered [ 6673.917806] LNet: 212723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6673.931959] LNetError: 212723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6673.954831] LNet: Removed LNI 192.168.201.126@tcp [ 6674.985075] Key type .llcrypt unregistered [ 6674.991172] Key type ._llcrypt unregistered [ 6701.097705] Key type ._llcrypt registered [ 6701.103901] Key type .llcrypt registered [ 6701.238774] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 6702.607324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6702.631738] alg: No test for adler32 (adler32-zlib) [ 6703.625737] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 6703.865417] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 6705.511154] Key type lgssc registered [ 6707.001899] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6715.997414] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6716.021833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6717.628871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6722.147632] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6727.136900] LustreError: 213990:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6727.151497] LustreError: 213990:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6731.132829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6731.457039] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6735.455444] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6742.671797] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6748.771385] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6755.919761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6756.219074] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6762.287926] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6763.381822] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6771.038509] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6788.075024] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6788.092803] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6788.107467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6789.094638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6789.097813] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6789.127425] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6791.847269] Lustre: server umount lustre-OST0000 complete [ 6798.314528] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6798.323454] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6798.336546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6798.339708] Lustre: Skipped 1 previous similar message [ 6801.874474] Lustre: server umount lustre-MDT0000 complete [ 6804.461978] LustreError: 216434:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6804.481424] LustreError: 216434:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6805.170184] LustreError: 213969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121894 with bad export cookie 544848983587628799 [ 6805.178461] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6805.179551] LustreError: 213969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6805.524912] Lustre: server umount lustre-MDT0001 complete [ 6816.082810] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 6819.191878] Key type lgssc unregistered [ 6819.463693] LNet: 217673:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6819.472417] LNetError: 217673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6819.487791] LNet: Removed LNI 192.168.201.126@tcp [ 6820.303844] Key type .llcrypt unregistered [ 6820.305440] Key type ._llcrypt unregistered [ 6839.484328] Key type ._llcrypt registered [ 6839.488716] Key type .llcrypt registered [ 6839.597897] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 6840.955970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6840.995042] alg: No test for adler32 (adler32-zlib) [ 6842.086913] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 6842.385100] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 6844.119135] Key type lgssc registered [ 6845.325423] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6852.478756] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6852.496500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6854.089059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6859.037564] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6864.354203] LustreError: 218945:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6864.365991] LustreError: 218945:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6868.188552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6868.762371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6873.367310] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6880.012724] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6886.261939] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6894.187483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6894.670245] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6899.829407] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6901.644055] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6910.268041] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6919.892293] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6919.941758] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6920.201605] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6927.031488] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6934.732627] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6936.472792] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6948.165565] Lustre: Permanently deactivating lustre-OST0001 [ 6948.190766] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6953.685722] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6965.563457] Lustre: Permanently reactivating lustre-OST0001 [ 6965.594369] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6965.600303] Lustre: Skipped 2 previous similar messages [ 6971.023841] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6971.040646] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6971.047698] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6971.060540] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6971.089101] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6976.480935] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6976.487089] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6976.489069] LustreError: Skipped 1 previous similar message [ 6976.490041] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6976.511237] Lustre: Skipped 2 previous similar messages [ 6978.843789] Lustre: server umount lustre-OST0001 complete [ 6982.153126] LustreError: 220770:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6982.178649] LustreError: 220770:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6985.186703] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6985.194264] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6985.212289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6985.215858] Lustre: Skipped 1 previous similar message [ 6987.232656] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6987.236415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6987.236893] LustreError: 222078:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6987.253424] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6987.325682] Lustre: Skipped 1 previous similar message [ 6989.182403] Lustre: server umount lustre-OST0000 complete [ 6996.974960] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6996.986477] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6997.016179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7000.053708] Lustre: server umount lustre-MDT0000 complete [ 7002.611659] LustreError: 219674:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7002.642845] LustreError: 219674:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7005.786356] LustreError: 218927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787122094 with bad export cookie 17042979941982548141 [ 7005.807452] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7005.815421] LustreError: 218927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7006.611195] Lustre: server umount lustre-MDT0001 complete [ 7019.245511] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 7022.344305] Key type lgssc unregistered [ 7022.606946] LNet: 223988:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7022.611708] LNetError: 223988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7022.643294] LNet: Removed LNI 192.168.201.126@tcp [ 7023.617695] Key type .llcrypt unregistered [ 7023.621375] Key type ._llcrypt unregistered [ 7063.399895] Key type ._llcrypt registered [ 7063.402462] Key type .llcrypt registered [ 7063.504611] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 7065.035459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7065.198051] alg: No test for adler32 (adler32-zlib) [ 7066.217961] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 7066.391394] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 7068.079248] Key type lgssc registered [ 7069.199953] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7079.653729] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7079.671181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7081.064957] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7081.097983] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7081.360681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7088.054544] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7097.407468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7097.549723] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7097.596542] Lustre: 226187:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7097.896707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7102.844486] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7110.958766] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7117.859606] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7126.098217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7126.431128] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7126.667313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7132.334209] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7133.057672] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7141.955420] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7146.668459] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7151.437520] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7157.146454] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7163.862814] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7179.021324] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7179.026298] Lustre: Skipped 1 previous similar message [ 7196.640476] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7196.645107] Lustre: Skipped 1 previous similar message [ 7238.445206] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7238.450533] Lustre: Skipped 4 previous similar messages [ 7308.940547] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 7308.945105] Lustre: Skipped 8 previous similar messages [ 7316.451901] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7316.462710] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7316.493752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7316.960329] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7322.276774] Lustre: server umount lustre-OST0000 complete [ 7327.202564] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7327.204365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7327.219138] Lustre: Skipped 2 previous similar messages [ 7327.237577] Lustre: Skipped 2 previous similar messages [ 7332.320942] LustreError: 226197:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7332.334859] LustreError: 226197:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7332.390817] Lustre: server umount lustre-MDT0000 complete [ 7336.103973] LustreError: 229471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787122425 with bad export cookie 3735981205158127610 [ 7336.114396] LustreError: 229471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7336.115053] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7336.350451] Lustre: server umount lustre-MDT0001 complete [ 7346.954610] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 7349.915477] Key type lgssc unregistered [ 7350.202738] LNet: 230971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7350.210354] LNetError: 230971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7351.270435] LNet: Removed LNI 192.168.201.126@tcp [ 7352.258064] Key type .llcrypt unregistered [ 7352.260828] Key type ._llcrypt unregistered [ 7372.877941] Key type ._llcrypt registered [ 7372.880522] Key type .llcrypt registered [ 7372.994359] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 7373.792987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7373.820083] alg: No test for adler32 (adler32-zlib) [ 7374.892727] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 7375.174485] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 7376.857433] Key type lgssc registered [ 7378.171199] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7385.906808] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7385.920121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7387.310724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7391.561655] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7397.869246] LustreError: 232365:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7397.878374] LustreError: 232365:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7401.228868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7402.144455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7408.416315] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7419.703518] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7425.984918] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7434.995641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7435.720610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7441.622451] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7452.145059] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7457.218720] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7477.728109] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7477.730182] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7477.744753] LustreError: Skipped 1 previous similar message [ 7477.772245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7477.782228] Lustre: Skipped 1 previous similar message [ 7477.811366] Lustre: Skipped 1 previous similar message [ 7480.106738] Lustre: server umount lustre-OST0000 complete [ 7487.970097] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7487.971293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7487.989524] Lustre: Skipped 1 previous similar message [ 7488.011133] Lustre: Skipped 1 previous similar message [ 7490.611534] Lustre: server umount lustre-MDT0000 complete [ 7493.088510] LustreError: 232361:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7493.125242] LustreError: 232361:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7494.690059] LustreError: 232345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787122583 with bad export cookie 13855810029115950871 [ 7494.696630] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7494.712583] LustreError: 232345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7495.577519] Lustre: server umount lustre-MDT0001 complete [ 7506.259572] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 7508.782956] Key type lgssc unregistered [ 7508.979574] LNet: 236062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7508.982774] LNetError: 236062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7508.991566] LNet: Removed LNI 192.168.201.126@tcp [ 7509.659571] Key type .llcrypt unregistered [ 7509.660770] Key type ._llcrypt unregistered [ 7535.899645] Key type ._llcrypt registered [ 7535.910505] Key type .llcrypt registered [ 7536.226939] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 7537.998327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7538.140184] alg: No test for adler32 (adler32-zlib) [ 7539.125501] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 7539.465033] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 7541.183172] Key type lgssc registered [ 7542.516593] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7814.266884] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 7817.158409] Key type lgssc unregistered [ 7817.428444] LNet: 238099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7817.438439] LNetError: 238099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7817.472868] LNet: Removed LNI 192.168.201.126@tcp [ 7818.532574] Key type .llcrypt unregistered [ 7818.538058] Key type ._llcrypt unregistered [ 7844.770955] Key type ._llcrypt registered [ 7844.774309] Key type .llcrypt registered [ 7844.881679] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 7860.976350] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 7862.138401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7862.194812] alg: No test for adler32 (adler32-zlib) [ 7863.238918] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 7863.590709] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 7865.327607] Key type lgssc registered [ 7866.901201] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7899.605984] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7583 sec ========== 03:03:06 (1787122986) [ 7902.110763] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:03:08 (1787122988) === [ 7905.984435] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:03:13 (1787122993) === [ 7955.126274] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 7958.956712] Key type lgssc unregistered [ 7959.444329] LNet: 243339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7959.447573] LNetError: 243339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7959.464854] LNet: Removed LNI 192.168.201.126@tcp [ 7960.507729] Key type .llcrypt unregistered [ 7960.511414] Key type ._llcrypt unregistered