[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 629425596 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003356] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010133] LSM: Security Framework initializing [ 0.011053] Yama: becoming mindful. [ 0.012036] SELinux: Initializing. [ 0.014062] *** VALIDATE selinux *** [ 0.025806] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.030686] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.031147] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033113] *** VALIDATE tmpfs *** [ 0.035113] *** VALIDATE proc *** [ 0.037027] *** VALIDATE cgroup *** [ 0.039008] *** VALIDATE cgroup2 *** [ 0.041367] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.042280] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.043011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.044032] Spectre V2 : User space: Vulnerable [ 0.045014] Speculative Store Bypass: Vulnerable [ 0.049005] debug: unmapping init [mem 0xffffffff9de59000-0xffffffff9de60fff] [ 0.052255] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.053772] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.054024] ... version: 2 [ 0.055014] ... bit width: 48 [ 0.056014] ... generic registers: 4 [ 0.057016] ... value mask: 0000ffffffffffff [ 0.058043] ... max period: 00007fffffffffff [ 0.059014] ... fixed-purpose events: 3 [ 0.060013] ... event mask: 000000070000000f [ 0.061335] rcu: Hierarchical SRCU implementation. [ 0.063685] smp: Bringing up secondary CPUs ... [ 0.064743] x86: Booting SMP configuration: [ 0.065026] .... node #0, CPUs: #1 #2 #3 [ 0.075092] smp: Brought up 1 node, 4 CPUs [ 0.077012] smpboot: Max logical packages: 1 [ 0.078016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.107193] node 0 deferred pages initialised in 27ms [ 0.109524] devtmpfs: initialized [ 0.110307] x86/mm: Memory block size: 128MB [ 0.114202] gcov: version magic: 0x41383552 [ 0.117394] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.120088] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.123300] pinctrl core: initialized pinctrl subsystem [ 0.126184] [ 0.126865] ************************************************************* [ 0.129015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.132013] ** ** [ 0.135012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.136010] ** ** [ 0.138013] ** This means that this kernel is built to expose internal ** [ 0.141012] ** IOMMU data structures, which may compromise security on ** [ 0.144014] ** your system. ** [ 0.146010] ** ** [ 0.148013] ** If you see this message and you are not debugging the ** [ 0.151013] ** kernel, report this immediately to your vendor! ** [ 0.153012] ** ** [ 0.155013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.158016] ************************************************************* [ 0.160815] NET: Registered protocol family 16 [ 0.162569] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.165054] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.168057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.172832] cpuidle: using governor menu [ 0.173889] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.176580] PCI: Using configuration type 1 for base access [ 0.179187] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.187201] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.189043] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.193312] cryptd: max_cpu_qlen set to 1000 [ 0.194787] ACPI: Added _OSI(Module Device) [ 0.195037] ACPI: Added _OSI(Processor Device) [ 0.197015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.199014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.204555] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.211300] ACPI: Interpreter enabled [ 0.213061] ACPI: PM: (supports S0 S3 S4 S5) [ 0.215012] ACPI: Using IOAPIC for interrupt routing [ 0.217451] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.224861] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.239234] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.241103] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.244023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.247417] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.252509] acpiphp: Slot [2] registered [ 0.254123] acpiphp: Slot [5] registered [ 0.255113] acpiphp: Slot [6] registered [ 0.257104] acpiphp: Slot [7] registered [ 0.258124] acpiphp: Slot [8] registered [ 0.260105] acpiphp: Slot [9] registered [ 0.261090] acpiphp: Slot [10] registered [ 0.262109] acpiphp: Slot [3] registered [ 0.264085] acpiphp: Slot [4] registered [ 0.265071] acpiphp: Slot [11] registered [ 0.268148] acpiphp: Slot [12] registered [ 0.269190] acpiphp: Slot [13] registered [ 0.271103] acpiphp: Slot [14] registered [ 0.272216] acpiphp: Slot [15] registered [ 0.274115] acpiphp: Slot [16] registered [ 0.276116] acpiphp: Slot [17] registered [ 0.280338] acpiphp: Slot [18] registered [ 0.283197] acpiphp: Slot [19] registered [ 0.285142] acpiphp: Slot [20] registered [ 0.287132] acpiphp: Slot [21] registered [ 0.288086] acpiphp: Slot [22] registered [ 0.289095] acpiphp: Slot [23] registered [ 0.291314] acpiphp: Slot [24] registered [ 0.294201] acpiphp: Slot [25] registered [ 0.295134] acpiphp: Slot [26] registered [ 0.296159] acpiphp: Slot [27] registered [ 0.298113] acpiphp: Slot [28] registered [ 0.299139] acpiphp: Slot [29] registered [ 0.300119] acpiphp: Slot [30] registered [ 0.302169] acpiphp: Slot [31] registered [ 0.303075] PCI host bridge to bus 0000:00 [ 0.304026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.307027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.309029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.311031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.313030] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.315031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.317225] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.320146] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.323438] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.332722] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.338017] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.340022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.342025] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.345017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.347017] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.350928] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.354059] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.358061] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.365132] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.383015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.391018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.397483] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.408016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.415015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.433015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.443466] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.459016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.470031] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.492143] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.504476] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.521016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.531017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.556023] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.572601] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.582016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.593015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.613018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.625598] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.637021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.644023] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.665021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.676362] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.687020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.693020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.711022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.722360] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.725518] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.728490] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.731487] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.734929] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.740133] iommu: Default domain type: Passthrough [ 0.743748] SCSI subsystem initialized [ 0.747331] ACPI: bus type USB registered [ 0.749156] usbcore: registered new interface driver usbfs [ 0.752202] usbcore: registered new interface driver hub [ 0.756126] usbcore: registered new device driver usb [ 0.764214] pps_core: LinuxPPS API ver. 1 registered [ 0.766023] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.769131] PTP clock support registered [ 0.773101] EDAC MC: Ver: 3.0.0 [ 0.775020] PCI: Using ACPI for IRQ routing [ 0.776167] NetLabel: Initializing [ 0.778011] NetLabel: domain hash size = 128 [ 0.780013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.783163] NetLabel: unlabeled traffic allowed by default [ 0.785125] vgaarb: loaded [ 0.787041] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.789011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.794598] clocksource: Switched to clocksource kvm-clock [ 0.928797] VFS: Disk quotas dquot_6.6.0 [ 0.930268] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.963949] *** VALIDATE ramfs *** [ 0.965549] *** VALIDATE hugetlbfs *** [ 0.967549] pnp: PnP ACPI init [ 0.969988] pnp: PnP ACPI: found 6 devices [ 0.993668] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.996367] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.999176] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.001840] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.005076] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.007153] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.009873] NET: Registered protocol family 2 [ 1.012417] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.020455] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.026208] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.033212] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.038724] TCP: Hash tables configured (established 65536 bind 65536) [ 1.043098] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.046797] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.050024] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.053515] NET: Registered protocol family 1 [ 1.056209] RPC: Registered named UNIX socket transport module. [ 1.058849] RPC: Registered udp transport module. [ 1.060892] RPC: Registered tcp transport module. [ 1.063068] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.066362] NET: Registered protocol family 44 [ 1.069475] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.071954] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.074423] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.077110] PCI: CLS 0 bytes, default 64 [ 1.078843] Unpacking initramfs... [ 2.720821] debug: unmapping init [mem 0xffff90247cc54000-0xffff90247ffbffff] [ 2.725393] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.727353] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.730502] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.249190] Initialise system trusted keyrings [ 3.250995] Key type blacklist registered [ 3.252777] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.262949] zbud: loaded [ 3.266106] *** VALIDATE nfs *** [ 3.267452] *** VALIDATE nfs4 *** [ 3.269226] pstore: using deflate compression [ 3.273386] Platform Keyring initialized [ 3.388913] NET: Registered protocol family 38 [ 3.391889] Key type asymmetric registered [ 3.393864] Asymmetric key parser 'x509' registered [ 3.395948] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.399663] io scheduler mq-deadline registered [ 3.401669] io scheduler kyber registered [ 3.403535] io scheduler bfq registered [ 3.405823] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.409323] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.412883] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.416116] ACPI: Power Button [PWRF] [ 3.427840] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.434832] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.457790] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.466810] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.488717] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.522009] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.550378] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.555312] Non-volatile memory driver v1.3 [ 3.557173] Linux agpgart interface v0.103 [ 3.587783] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.590801] vda: detected capacity change from 0 to 74698752 [ 3.617104] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.623360] vdb: detected capacity change from 0 to 1073741824 [ 3.651920] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.656839] vdc: detected capacity change from 0 to 2621440000 [ 3.676828] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.680034] vdd: detected capacity change from 0 to 2621440000 [ 3.695893] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.699246] vde: detected capacity change from 0 to 4294967296 [ 3.715501] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.719018] vdf: detected capacity change from 0 to 4294967296 [ 3.727578] libphy: Fixed MDIO Bus: probed [ 3.738670] usbcore: registered new interface driver usbserial_generic [ 3.741863] usbserial: USB Serial support registered for generic [ 3.745461] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.753689] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.756032] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.758705] mousedev: PS/2 mouse device common for all mice [ 3.762985] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.765682] rtc_cmos 00:05: RTC can wake from S4 [ 3.774301] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.774678] rtc_cmos 00:05: registered as rtc0 [ 3.782414] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.785327] intel_pstate: CPU model not supported [ 3.785734] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.793749] hid: raw HID events driver (C) Jiri Kosina [ 3.798965] usbcore: registered new interface driver usbhid [ 3.802256] usbhid: USB HID core driver [ 3.805545] drop_monitor: Initializing network drop monitor service [ 3.810105] Initializing XFRM netlink socket [ 3.813941] NET: Registered protocol family 10 [ 3.818965] Segment Routing with IPv6 [ 3.820527] NET: Registered protocol family 17 [ 3.823069] mpls_gso: MPLS GSO support [ 3.829058] RAS: Correctable Errors collector initialized. [ 3.832272] AVX version of gcm_enc/dec engaged. [ 3.834419] AES CTR mode by8 optimization enabled [ 3.945422] sched_clock: Marking stable (3945380910, 0)->(5095165559, -1149784649) [ 3.952355] registered taskstats version 1 [ 3.957723] Loading compiled-in X.509 certificates [ 3.961383] zswap: loaded using pool lzo/zbud [ 3.992714] Key type big_key registered [ 4.008503] Key type encrypted registered [ 4.010644] ima: No TPM chip found, activating TPM-bypass! [ 4.013627] ima: Allocated hash algorithm: sha1 [ 4.015570] ima: No architecture policies found [ 4.019902] evm: Initialising EVM extended attributes: [ 4.021915] evm: security.selinux [ 4.023108] evm: security.ima [ 4.024289] evm: security.capability [ 4.025604] evm: HMAC attrs: 0x1 [ 4.028593] rtc_cmos 00:05: setting system clock to 2026-08-08 08:23:05 UTC (1786177385) [ 4.037786] debug: unmapping init [mem 0xffffffff9ee03000-0xffffffff9effffff] [ 4.050275] debug: unmapping init [mem 0xffffffff9db82000-0xffffffff9de58fff] [ 4.060217] Write protecting the kernel read-only data: 28672k [ 4.064930] debug: unmapping init [mem 0xffffffff9c203000-0xffffffff9c3fffff] [ 4.069346] debug: unmapping init [mem 0xffffffff9cb14000-0xffffffff9cbfffff] [ 4.121247] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.130876] systemd[1]: Detected virtualization kvm. [ 4.133421] systemd[1]: Detected architecture x86-64. [ 4.135671] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.163883] systemd[1]: No hostname configured. [ 4.165750] systemd[1]: Set hostname to . [ 4.167857] random: systemd: uninitialized urandom read (16 bytes read) [ 4.170743] systemd[1]: Initializing machine ID from random generator. [ 4.254939] random: ln: uninitialized urandom read (6 bytes read) [ 4.433810] random: systemd: uninitialized urandom read (16 bytes read) [ 4.437929] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.455507] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 4.464757] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Reached target Swap. Starting Setup Virtual Console... [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.865295] device-mapper: uevent: version 1.0.3 [ 5.867719] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 7.705764] random: fast init done [ 7.759977] virtio_net virtio0 ens2: renamed from eth0 [ 8.016620] scsi host0: ata_piix [ 8.055651] scsi host1: ata_piix [ 8.061580] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.064131] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 12.722956] random: crng init done [ 12.728537] random: 7 urandom warning(s) missed due to ratelimiting [ 13.398830] dracut-initqueue[593]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 15.086234] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 17.848587] printk: systemd: 26 output lines suppressed due to ratelimiting [ 18.768744] SELinux: Disabled at runtime. [ 18.840718] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.854333] systemd[1]: Detected virtualization kvm. [ 18.857138] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 20.132935] systemd[1]: initrd-switch-root.service: Succeeded. [ 20.138771] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 20.156856] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 20.167259] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 20.169310] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 20.180874] systemd[1]: Starting Journal Service... Starting Journal Service... [ 20.446854] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Kernel Debug File System... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Paths. [ 22.701344] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 24.328304] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 25.338281] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 25.573679] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 26.514652] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 26.868814] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…only root support (10s / no limit) [** ] A start job is running for Configur…only root support (10s / no limit)[ 30.954244] Key type dns_resolver registered [*** ] A start job is running for Configur…only root support (11s / no limit)[ 31.554959] NFS: Registering the id_resolver key type [ 31.557964] Key type id_resolver registered [ 31.559851] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (11s / no limit) [ *** ] A start job is running for Configur…only root support (12s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg333-server login: [ 61.695011] hrtimer: interrupt took 2003904 ns [ 95.666157] libcfs: loading out-of-tree module taints kernel. [ 95.700716] Key type ._llcrypt registered [ 95.703120] Key type .llcrypt registered [ 95.892709] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 114.953784] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 117.554142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 117.569426] alg: No test for adler32 (adler32-zlib) [ 118.977369] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 119.986580] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 121.703163] Key type lgssc registered [ 122.979240] Lustre: Echo OBD driver; http://www.lustre.org/ [ 141.996973] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 182.046737] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 198.308454] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 198.339325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 199.578045] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 199.611637] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 199.678250] Lustre: lustre-MDT0000: new disk, initializing [ 199.755712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 199.775696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 205.083886] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 217.790243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 217.879475] Lustre: 6520:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 217.905193] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 217.909540] Lustre: Skipped 1 previous similar message [ 218.058299] Lustre: lustre-MDT0001: new disk, initializing [ 218.175701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 218.225342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 218.240296] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 222.551845] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 227.917693] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 237.026561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 237.259878] Lustre: lustre-OST0000: new disk, initializing [ 237.263394] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 237.271175] Lustre: 8459:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 237.366155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 241.371555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 241.384818] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 241.437234] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 242.965388] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 257.468040] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 257.655264] Lustre: lustre-OST0001: new disk, initializing [ 257.671776] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 257.683450] Lustre: 9532:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 257.758406] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 263.732439] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 265.252869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 265.266630] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 265.360494] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 277.173545] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 285.938881] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 291.782417] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing check_logdir /tmp/testlogs/ [ 296.826648] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing yml_node [ 302.401332] Lustre: DEBUG MARKER: Client: 2.17.56.50 [ 304.586311] Lustre: DEBUG MARKER: MDS: 2.17.56.50 [ 307.193268] Lustre: DEBUG MARKER: OSS: 2.17.56.50 [ 309.485327] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 8 04:28:08 EDT 2026 [ 324.913506] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 327.069740] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 342.046254] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 342.055703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 342.061211] Lustre: Skipped 3 previous similar messages [ 342.107246] Lustre: Skipped 3 previous similar messages [ 347.136211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 347.143486] Lustre: Skipped 3 previous similar messages [ 348.293094] Lustre: server umount lustre-MDT0000 complete [ 351.873619] LustreError: 6514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786177733 with bad export cookie 13096137468559345025 [ 351.876961] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 351.899344] LustreError: 6514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 352.246785] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 352.296595] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 352.312022] Lustre: Skipped 1 previous similar message [ 356.259272] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 356.270712] Lustre: Skipped 1 previous similar message [ 358.594941] Lustre: server umount lustre-MDT0001 complete [ 362.622743] Lustre: server umount lustre-OST0000 complete [ 365.869409] Lustre: server umount lustre-OST0001 complete [ 371.948518] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 379.499302] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 415.081816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 415.284177] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 415.303220] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 415.356712] Lustre: lustre-MDT0000: new disk, initializing [ 415.414445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 415.427937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 420.202407] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 430.086534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 430.263549] Lustre: 15084:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 430.462541] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 430.471800] Lustre: Skipped 1 previous similar message [ 430.553426] Lustre: lustre-MDT0001: new disk, initializing [ 430.638159] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 430.678021] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 430.691048] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 435.375600] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 445.183557] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 450.548520] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 457.265640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 457.589356] Lustre: lustre-OST0000: new disk, initializing [ 457.594625] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 457.605766] Lustre: 17052:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 457.694580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 459.368204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 459.376514] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 459.507509] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 463.330754] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 472.268649] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 476.970569] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 477.179409] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 481.016762] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 481.321261] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 485.347466] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 485.348139] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 485.366610] Lustre: Skipped 1 previous similar message [ 485.371147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 489.468348] Lustre: server umount lustre-OST0000 complete [ 495.587585] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 495.596433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 495.646164] Lustre: Skipped 2 previous similar messages [ 507.359128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 507.568192] Lustre: server umount lustre-MDT0000 complete [ 509.929035] LustreError: 15096:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 509.953366] LustreError: 15096:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 6 previous similar messages [ 511.074297] LustreError: 15076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786177892 with bad export cookie 13096137468559346558 [ 511.100133] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 511.112173] LustreError: 15076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 511.818355] Lustre: server umount lustre-MDT0001 complete [ 515.119313] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 04:31:34 (1786177894) [ 521.657829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 522.024588] LustreError: 19319:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 522.124652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 526.610619] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 527.384248] LustreError: 19320:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 532.469050] LustreError: 19319:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 535.829399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 540.468826] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 548.205829] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 554.043303] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 561.056733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 567.430112] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 575.751794] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 586.209301] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 586.211959] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 586.291409] Lustre: Skipped 1 previous similar message [ 586.292679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 586.326463] LustreError: Skipped 1 previous similar message [ 586.370106] Lustre: Skipped 4 previous similar messages [ 587.140975] Lustre: server umount lustre-OST0000 complete [ 605.152220] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 605.273462] Lustre: server umount lustre-MDT0000 complete [ 605.672258] LustreError: 19320:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 605.710823] LustreError: 19320:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 608.884637] LustreError: 19302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786177990 with bad export cookie 13096137468559347440 [ 608.891770] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 608.892470] LustreError: 19302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 609.134677] Lustre: server umount lustre-MDT0001 complete [ 620.867885] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 623.152250] Key type lgssc unregistered [ 623.482958] LNet: 22856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 623.490224] LNetError: 22856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 623.537360] LNet: Removed LNI 192.168.203.133@tcp [ 624.546147] Key type .llcrypt unregistered [ 624.554379] Key type ._llcrypt unregistered [ 647.085717] Key type ._llcrypt registered [ 647.088737] Key type .llcrypt registered [ 647.207718] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 648.283308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 648.360992] alg: No test for adler32 (adler32-zlib) [ 649.348754] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 649.565414] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 651.191150] Key type lgssc registered [ 652.137896] Lustre: Echo OBD driver; http://www.lustre.org/ [ 659.372959] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 659.397145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 660.850949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 665.093163] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 671.202180] LustreError: 24262:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 671.207725] LustreError: 24262:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 673.338254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 673.583249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 678.298980] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 687.906849] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 693.599922] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 700.229424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 700.523422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 705.655915] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 707.549667] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 715.294963] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 731.618496] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 731.639355] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 731.729681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 733.665908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 733.668851] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 733.680920] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 735.164807] Lustre: server umount lustre-OST0000 complete [ 740.353724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 740.357611] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 740.365147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 740.368657] Lustre: Skipped 1 previous similar message [ 743.905673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 743.906039] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 746.244818] Lustre: server umount lustre-MDT0000 complete [ 749.028470] LustreError: 24258:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 749.063799] LustreError: 24258:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 751.592601] LustreError: 24242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786178133 with bad export cookie 10716604251177441900 [ 751.600385] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 751.614672] LustreError: 24242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 752.124975] Lustre: server umount lustre-MDT0001 complete [ 762.691966] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 765.043851] Key type lgssc unregistered [ 765.363779] LNet: 28244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 765.385221] LNetError: 28244:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 765.396911] LNet: Removed LNI 192.168.203.133@tcp [ 766.503502] Key type .llcrypt unregistered [ 766.506700] Key type ._llcrypt unregistered [ 791.808262] Key type ._llcrypt registered [ 791.810309] Key type .llcrypt registered [ 791.881995] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 792.596540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 792.672810] alg: No test for adler32 (adler32-zlib) [ 793.789582] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 793.955821] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 795.631230] Key type lgssc registered [ 796.936177] Lustre: Echo OBD driver; http://www.lustre.org/ [ 805.038457] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 805.051686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 806.306252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 810.611535] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 815.664024] LustreError: 29659:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 815.699647] LustreError: 29659:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 819.104928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 819.362857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 823.883724] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 832.030574] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 838.329969] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 851.968575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 852.308687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 858.467903] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 866.211736] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 875.925889] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 885.246538] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 885.250780] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 885.257446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 886.242028] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 886.252745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 886.256598] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 888.809700] Lustre: server umount lustre-OST0000 complete [ 896.481825] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 896.490870] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 896.516680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 896.521700] Lustre: Skipped 1 previous similar message [ 896.532264] Lustre: Skipped 2 previous similar messages [ 898.814097] Lustre: server umount lustre-MDT0000 complete [ 901.600819] LustreError: 30386:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 901.612975] LustreError: 30386:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 902.126415] LustreError: 29638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786178283 with bad export cookie 12070396250730333578 [ 902.128235] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 902.134390] LustreError: 29638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 902.411847] Lustre: server umount lustre-MDT0001 complete [ 914.395707] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 916.752901] Key type lgssc unregistered [ 916.953695] LNet: 33563:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 916.957329] LNetError: 33563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 917.050767] LNet: Removed LNI 192.168.203.133@tcp [ 917.921870] Key type .llcrypt unregistered [ 917.923715] Key type ._llcrypt unregistered [ 942.680270] Key type ._llcrypt registered [ 942.684651] Key type .llcrypt registered [ 942.882541] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 944.912744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 945.074966] alg: No test for adler32 (adler32-zlib) [ 946.083644] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 946.351371] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 948.384068] Key type lgssc registered [ 949.747265] Lustre: Echo OBD driver; http://www.lustre.org/ [ 957.547299] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 957.558628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 958.909908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 963.152740] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 969.189731] LustreError: 34975:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 969.202483] LustreError: 34975:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 970.160881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 970.498397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 974.555418] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 981.790605] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 987.634663] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 994.444312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 994.691449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1000.186717] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1003.973121] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1007.864908] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1016.288695] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1016.301665] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1016.326664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1018.073866] Lustre: server umount lustre-OST0000 complete [ 1021.922164] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1021.946215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1021.980901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1024.481490] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1024.481977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1024.494066] Lustre: Skipped 1 previous similar message [ 1029.604955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1029.611074] Lustre: Skipped 1 previous similar message [ 1031.715690] Lustre: server umount lustre-MDT0000 complete [ 1033.698389] LustreError: 35703:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1033.751231] LustreError: 35703:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 1035.707712] LustreError: 34957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786178417 with bad export cookie 2201705294074333810 [ 1035.712872] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1035.717565] LustreError: 34957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1035.964986] Lustre: server umount lustre-MDT0001 complete [ 1046.768422] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 1048.719573] Key type lgssc unregistered [ 1049.136540] LNet: 38512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1049.150672] LNetError: 38512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1049.163462] LNet: Removed LNI 192.168.203.133@tcp [ 1049.729499] Key type .llcrypt unregistered [ 1049.732440] Key type ._llcrypt unregistered [ 1071.746701] Key type ._llcrypt registered [ 1071.749364] Key type .llcrypt registered [ 1071.839903] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 1072.844101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1072.866622] alg: No test for adler32 (adler32-zlib) [ 1074.121601] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 1074.333576] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 1076.087309] Key type lgssc registered [ 1077.092636] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1084.864400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1084.878106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1086.278899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1090.823444] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1095.650454] LustreError: 39904:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1095.659596] LustreError: 39904:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 1098.587511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1098.840546] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1103.922442] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1110.931227] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1116.748710] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1123.666816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1124.195822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1131.303937] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1131.380575] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1138.663628] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1145.832728] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1145.842685] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1145.850965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1146.570088] Lustre: lustre-OST0000: Not available for connect from 192.168.203.33@tcp (stopping) [ 1146.913357] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1151.498281] Lustre: server umount lustre-OST0000 complete [ 1170.419278] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1170.430292] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1170.448690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1170.451599] Lustre: Skipped 2 previous similar messages [ 1172.469940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1172.472070] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1176.281287] Lustre: server umount lustre-MDT0000 complete [ 1178.592291] LustreError: 40631:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1178.704530] LustreError: 40631:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 1180.324645] LustreError: 40638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786178561 with bad export cookie 15848096374223363609 [ 1180.332866] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1180.336459] LustreError: 40638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1186.638820] Lustre: server umount lustre-MDT0001 complete [ 1196.859456] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 1199.576135] Key type lgssc unregistered [ 1199.964063] LNet: 43614:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1199.971375] LNetError: 43614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1200.996857] LNet: Removed LNI 192.168.203.133@tcp [ 1201.710937] Key type .llcrypt unregistered [ 1201.718143] Key type ._llcrypt unregistered [ 1224.428857] Key type ._llcrypt registered [ 1224.431155] Key type .llcrypt registered [ 1224.522599] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 1225.723207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1225.921629] alg: No test for adler32 (adler32-zlib) [ 1227.122520] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 1227.407062] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 1229.069909] Key type lgssc registered [ 1230.173172] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1237.174712] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1237.212490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1238.589603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1242.747115] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1248.736827] LustreError: 45026:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1248.767620] LustreError: 45026:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 1249.219414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1249.453532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1253.252756] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1260.343276] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1265.368892] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1271.426689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1271.663871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1278.017052] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1284.047523] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1284.562450] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1292.257624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1292.266857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1292.269213] Lustre: Skipped 2 previous similar messages [ 1292.283894] Lustre: Skipped 2 previous similar messages [ 1293.025435] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.33@tcp (stopping) [ 1293.730677] Lustre: server umount lustre-MDT0000 complete [ 1297.311044] LustreError: 45007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786178678 with bad export cookie 12731514162076231822 [ 1297.316650] LustreError: 45007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1297.318350] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1297.376644] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1297.382189] LustreError: 45021:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1297.386165] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1302.497232] LustreError: 45023:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1302.498284] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1302.516912] LustreError: 45023:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 1302.524574] Lustre: Skipped 1 previous similar message [ 1303.579969] Lustre: server umount lustre-MDT0001 complete [ 1332.191392] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1332.317552] Lustre: server umount lustre-OST0000 complete [ 1349.680319] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 1351.991796] Key type lgssc unregistered [ 1352.245604] LNet: 48890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1352.259514] LNetError: 48890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1352.268944] LNet: Removed LNI 192.168.203.133@tcp [ 1353.063978] Key type .llcrypt unregistered [ 1353.066864] Key type ._llcrypt unregistered [ 1378.919359] Key type ._llcrypt registered [ 1378.921743] Key type .llcrypt registered [ 1379.099369] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 1380.854830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1381.501994] alg: No test for adler32 (adler32-zlib) [ 1382.611557] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 1382.802426] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 1384.488112] Key type lgssc registered [ 1385.625243] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1392.856232] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1392.882942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1408.543396] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1413.727844] LustreError: 50278:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.133@tcp: failed processing log, type 4: rc = -110 [ 1440.609631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1446.934873] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1454.741433] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1621.560819] Lustre: server umount lustre-OST0000 complete [ 1640.151430] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 1642.406438] Key type lgssc unregistered [ 1642.626557] LNet: 51959:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1642.631179] LNetError: 51959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1642.645927] LNet: Removed LNI 192.168.203.133@tcp [ 1643.484338] Key type .llcrypt unregistered [ 1643.486889] Key type ._llcrypt unregistered [ 1668.312482] Key type ._llcrypt registered [ 1668.314641] Key type .llcrypt registered [ 1668.409667] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 1669.573392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1669.708660] alg: No test for adler32 (adler32-zlib) [ 1670.894602] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 1671.133632] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 1672.760339] Key type lgssc registered [ 1674.124969] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1679.659764] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1679.698846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1681.211617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1686.541600] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1690.602887] LustreError: 53376:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1690.625128] LustreError: 53376:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 1695.182319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1695.554859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1699.459429] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1707.079274] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1713.239586] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1719.124290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1719.358176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1725.124667] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1726.529927] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1732.812836] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1736.673809] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1736.676987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1736.685382] Lustre: Skipped 1 previous similar message [ 1736.691904] Lustre: Skipped 1 previous similar message [ 1741.800703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1741.805992] Lustre: Skipped 1 previous similar message [ 1742.055472] Lustre: server umount lustre-OST0000 complete [ 1746.913988] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1746.914549] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1746.917106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1746.953752] Lustre: Skipped 1 previous similar message [ 1751.524191] Lustre: server umount lustre-MDT0000 complete [ 1752.036561] LustreError: 54412:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1752.066417] LustreError: 54412:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 1754.597467] LustreError: 53356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786179136 with bad export cookie 15622325518034065411 [ 1754.604463] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1754.608458] LustreError: 53356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1754.801777] Lustre: server umount lustre-MDT0001 complete [ 1763.992815] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 1767.266505] Key type lgssc unregistered [ 1767.582403] LNet: 56909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1767.589028] LNetError: 56909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1767.617271] LNet: Removed LNI 192.168.203.133@tcp [ 1768.296236] Key type .llcrypt unregistered [ 1768.299097] Key type ._llcrypt unregistered [ 1789.604478] Key type ._llcrypt registered [ 1789.620191] Key type .llcrypt registered [ 1789.775268] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 1790.966068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1791.016426] alg: No test for adler32 (adler32-zlib) [ 1792.480410] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 1792.731716] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 1794.399405] Key type lgssc registered [ 1795.315890] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1802.826624] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1802.863208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1818.335370] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1823.455467] LustreError: 58289:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.133@tcp: failed processing log, type 4: rc = -110 [ 1850.143206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1855.982109] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1862.756197] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1869.106754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1869.476778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1869.539091] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1873.454797] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1874.933023] LustreError: 58314:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1874.959496] LustreError: 58314:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 1880.037236] LustreError: 58314:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1881.656055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1882.039488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1886.330309] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1894.029463] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1899.495537] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1902.560826] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1902.572316] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1902.585362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1905.636497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1905.636521] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1905.643281] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1910.756581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1910.759821] Lustre: Skipped 2 previous similar messages [ 1915.875247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1915.879870] Lustre: Skipped 1 previous similar message [ 1916.408457] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1916.544438] Lustre: server umount lustre-OST0000 complete [ 1938.401211] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1938.410450] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1938.425729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1941.480960] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1943.726661] Lustre: server umount lustre-MDT0000 complete [ 1945.643114] LustreError: 59330:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1945.658856] LustreError: 59330:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 1947.737813] LustreError: 58296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786179329 with bad export cookie 7261161989190329392 [ 1947.746564] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1947.755075] LustreError: 58296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1954.135641] Lustre: server umount lustre-MDT0001 complete [ 1964.930758] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 1967.425544] Key type lgssc unregistered [ 1967.691714] LNet: 62035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1967.698651] LNetError: 62035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1967.713437] LNet: Removed LNI 192.168.203.133@tcp [ 1968.671874] Key type .llcrypt unregistered [ 1968.674823] Key type ._llcrypt unregistered [ 1990.050813] Key type ._llcrypt registered [ 1990.055709] Key type .llcrypt registered [ 1990.169214] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 1991.177333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1991.195367] alg: No test for adler32 (adler32-zlib) [ 1992.198676] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 1992.369179] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 1994.007167] Key type lgssc registered [ 1995.065825] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2002.519833] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2002.537391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2004.024601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2008.643207] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2014.184166] LustreError: 63428:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2014.238660] LustreError: 63428:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 2016.730515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2016.996634] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2021.618135] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2029.836751] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2035.529201] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2043.108476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2043.552361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2048.620412] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2050.141944] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2058.405578] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2077.668653] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2077.669153] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2077.672711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2077.689420] Lustre: Skipped 1 previous similar message [ 2077.928230] Lustre: server umount lustre-OST0000 complete [ 2082.785326] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2082.790330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2082.793743] Lustre: Skipped 1 previous similar message [ 2082.798774] Lustre: Skipped 1 previous similar message [ 2087.776837] Lustre: server umount lustre-MDT0000 complete [ 2087.910520] LustreError: 63429:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2087.928115] LustreError: 63429:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 2091.224470] LustreError: 63409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786179472 with bad export cookie 11809601992804583827 [ 2091.228993] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2091.234157] LustreError: 63409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2091.561499] Lustre: server umount lustre-MDT0001 complete [ 2103.133327] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 2105.544616] Key type lgssc unregistered [ 2105.792652] LNet: 66983:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2105.807401] LNetError: 66983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2106.854423] LNet: Removed LNI 192.168.203.133@tcp [ 2107.706737] Key type .llcrypt unregistered [ 2107.708993] Key type ._llcrypt unregistered [ 2135.045193] Key type ._llcrypt registered [ 2135.047272] Key type .llcrypt registered [ 2135.126926] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 2137.052082] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2137.086134] alg: No test for adler32 (adler32-zlib) [ 2138.052405] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 2138.363581] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 2140.151192] Key type lgssc registered [ 2141.106694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2151.795618] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 04:58:51 (1786179531) [ 2158.113570] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2158.126986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2159.490729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2163.245444] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2168.805595] LustreError: 68691:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2168.823277] LustreError: 68691:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 2172.633551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2172.855619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2177.423572] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2186.166566] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2192.429756] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2198.420803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2198.809740] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2203.959743] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2204.691706] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2211.893195] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2222.088188] Lustre: Failing over lustre-MDT0000 [ 2222.558244] Lustre: server umount lustre-MDT0000 complete [ 2223.256335] LustreError: 68685:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2223.587454] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2223.601431] Lustre: Skipped 1 previous similar message [ 2228.424175] LustreError: 68686:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2228.438967] LustreError: 68686:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 3 previous similar messages [ 2230.637819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2230.704775] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2230.815975] Lustre: *** cfs_fail_loc=135, val=0*** [ 2230.880426] LustreError: 71776:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2230.887053] LustreError: 71776:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.133@tcp: cfg command failed: rc = -2 [ 2230.895207] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2230.895207] [ 2230.902685] LustreError: MGC192.168.203.133@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2230.913037] LustreError: 71764:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2230.921457] LustreError: 71764:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2230.931806] LustreError: 71764:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2230.936170] LustreError: 71764:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2230.969847] Lustre: server umount lustre-MDT0000 complete [ 2230.975945] LustreError: 71764:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2233.534333] LustreError: 68685:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2233.594605] LustreError: 68685:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 3 previous similar messages [ 2238.398058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2238.466777] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2238.476196] LustreError: 68683:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2238.483852] LustreError: 72121:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.203.133@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (000000003a731396) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2238.503430] LustreError: 68691:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2238.513403] LustreError: 68691:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 5 previous similar messages [ 2238.647388] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.33@tcp (not set up) [ 2238.727028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2238.802685] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2243.144698] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2243.771129] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2244.075560] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2244.112857] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2244.139308] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2254.318933] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2254.324339] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2254.333664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2254.339179] Lustre: Skipped 1 previous similar message [ 2256.506052] Lustre: server umount lustre-OST0000 complete [ 2264.545975] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2264.547186] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2264.552784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2264.552793] Lustre: Skipped 1 previous similar message [ 2264.559659] Lustre: Skipped 1 previous similar message [ 2265.751977] Lustre: server umount lustre-MDT0000 complete [ 2269.311458] LustreError: 68671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786179650 with bad export cookie 13038661636338025489 [ 2269.324320] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2269.663990] Lustre: server umount lustre-MDT0001 complete [ 2281.206088] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 2283.943420] Key type lgssc unregistered [ 2284.167864] LNet: 73572:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2284.178209] LNetError: 73572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2284.194503] LNet: Removed LNI 192.168.203.133@tcp [ 2284.956829] Key type .llcrypt unregistered [ 2284.961168] Key type ._llcrypt unregistered [ 2309.858175] Key type ._llcrypt registered [ 2309.863433] Key type .llcrypt registered [ 2310.007613] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 2311.153694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2311.169188] alg: No test for adler32 (adler32-zlib) [ 2312.169612] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 2312.351837] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 2314.159485] Key type lgssc registered [ 2315.198799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2323.461425] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2323.493849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2324.769464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2329.610857] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2335.203268] LustreError: 74985:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2335.219448] LustreError: 74985:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 2336.230199] LustreError: 74986:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2337.597984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2337.912099] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2342.119258] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2348.750216] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2354.408886] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2361.726731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2361.961540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2368.530205] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2371.195162] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2375.710192] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2381.368527] Lustre: Failing over lustre-MDT0000 [ 2381.803997] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2381.816446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2381.844112] Lustre: Skipped 1 previous similar message [ 2381.865086] LustreError: 76038:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2381.998313] Lustre: server umount lustre-MDT0000 complete [ 2386.916244] LustreError: 74986:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2386.927537] LustreError: 74986:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 4 previous similar messages [ 2391.242481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2391.329283] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2391.432080] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2391.505584] LustreError: 78070:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2391.514565] LustreError: 78070:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.133@tcp: cfg command failed: rc = -9 [ 2391.519674] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2391.519674] [ 2391.526054] LustreError: MGC192.168.203.133@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2391.538877] LustreError: 78059:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2391.545358] LustreError: 78059:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2391.551655] LustreError: 78059:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2391.556649] LustreError: 78059:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2391.582958] Lustre: server umount lustre-MDT0000 complete [ 2391.591225] LustreError: 78059:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2392.033237] LustreError: 74985:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2392.045212] LustreError: 74985:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 3 previous similar messages [ 2397.101947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2397.179826] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2397.447892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2397.519161] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2398.451811] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2401.969799] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2402.811514] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2403.013037] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2403.113636] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2410.465367] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2410.481792] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2410.501112] Lustre: Skipped 1 previous similar message [ 2410.507383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2413.035892] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2413.043092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2413.092972] Lustre: Skipped 1 previous similar message [ 2414.792943] Lustre: server umount lustre-OST0000 complete [ 2418.144993] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2418.152456] Lustre: Skipped 1 previous similar message [ 2418.162359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2418.168699] Lustre: Skipped 1 previous similar message [ 2423.264338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2424.376500] Lustre: server umount lustre-MDT0000 complete [ 2428.293538] LustreError: 74968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786179809 with bad export cookie 15741985095583262991 [ 2428.306244] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2428.511626] Lustre: server umount lustre-MDT0001 complete [ 2440.541511] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 2443.122912] Key type lgssc unregistered [ 2443.369772] LNet: 79854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2443.379154] LNetError: 79854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2443.400447] LNet: Removed LNI 192.168.203.133@tcp [ 2444.330577] Key type .llcrypt unregistered [ 2444.335920] Key type ._llcrypt unregistered [ 2458.456082] Key type ._llcrypt registered [ 2458.458432] Key type .llcrypt registered [ 2458.533611] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 2458.825488] Key type .llcrypt unregistered [ 2458.832755] Key type ._llcrypt unregistered [ 2482.689644] Key type ._llcrypt registered [ 2482.691645] Key type .llcrypt registered [ 2482.769526] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 2483.507896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2483.517591] alg: No test for adler32 (adler32-zlib) [ 2484.540718] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 2484.745054] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 2486.407517] Key type lgssc registered [ 2487.439560] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2494.978935] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2495.034638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2496.458649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2500.602554] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2505.773764] LustreError: 81763:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2505.808698] LustreError: 81763:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 2509.058957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2509.340412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2513.582082] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2521.157462] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2526.333747] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2532.971810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2533.232313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2538.375344] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2541.429206] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2545.674343] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2555.875701] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2555.894386] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2555.911578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2556.902465] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2556.909167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2556.956705] Lustre: Skipped 1 previous similar message [ 2560.369632] Lustre: server umount lustre-OST0000 complete [ 2565.684690] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2565.699681] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2565.712455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2569.995478] Lustre: server umount lustre-MDT0000 complete [ 2572.259775] LustreError: 81757:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2572.280941] LustreError: 81757:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 2573.257253] LustreError: 81742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786179954 with bad export cookie 4087994474067068146 [ 2573.260656] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2573.268861] LustreError: 81742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2573.628758] Lustre: server umount lustre-MDT0001 complete [ 2586.783678] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 2589.650694] Key type lgssc unregistered [ 2590.135488] LNet: 85300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2590.150871] LNetError: 85300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2591.208899] LNet: Removed LNI 192.168.203.133@tcp [ 2592.277561] Key type .llcrypt unregistered [ 2592.280064] Key type ._llcrypt unregistered [ 2615.063110] Key type ._llcrypt registered [ 2615.065846] Key type .llcrypt registered [ 2615.144485] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 2616.360168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2616.459715] alg: No test for adler32 (adler32-zlib) [ 2617.480358] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 2617.707335] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 2619.359494] Key type lgssc registered [ 2620.286741] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2626.511387] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2626.522573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2627.942224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2632.012231] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2638.311207] LustreError: 86693:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2638.329126] LustreError: 86693:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 2639.387082] LustreError: 86694:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2641.105522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2641.438475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2645.966175] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2653.163309] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2658.991387] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2665.831970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2666.105659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2671.473909] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2674.366775] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2677.751955] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2684.905513] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2684.914376] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2684.931531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2684.947525] Lustre: Skipped 1 previous similar message [ 2684.981265] Lustre: Skipped 1 previous similar message [ 2686.167910] Lustre: server umount lustre-OST0000 complete [ 2690.030592] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2690.044645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2690.049836] Lustre: Skipped 1 previous similar message [ 2690.057366] Lustre: Skipped 1 previous similar message [ 2695.142676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2695.154157] Lustre: Skipped 1 previous similar message [ 2695.566471] Lustre: server umount lustre-MDT0000 complete [ 2699.996379] LustreError: 86674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786180081 with bad export cookie 12309724314390269328 [ 2700.020867] LustreError: 86674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2700.021936] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2700.439916] Lustre: server umount lustre-MDT0001 complete [ 2711.953244] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 2714.061669] Key type lgssc unregistered [ 2714.341712] LNet: 90228:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2714.346913] LNetError: 90228:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2715.369782] LNet: Removed LNI 192.168.203.133@tcp [ 2716.022919] Key type .llcrypt unregistered [ 2716.025096] Key type ._llcrypt unregistered [ 2738.756560] Key type ._llcrypt registered [ 2738.758296] Key type .llcrypt registered [ 2738.812226] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 2739.450458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2739.492552] alg: No test for adler32 (adler32-zlib) [ 2740.452761] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 2740.633852] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 2742.327465] Key type lgssc registered [ 2743.603866] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2750.906660] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2750.919252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2752.295493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2756.404671] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2761.717650] LustreError: 91642:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2761.731807] LustreError: 91642:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 2764.750748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2765.154964] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2769.946361] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2777.412708] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2782.865818] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2789.728079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2789.947307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2795.824395] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2797.099760] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2804.187779] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2816.514706] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2816.524054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2816.541405] LustreError: 93465:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2816.827790] Lustre: server umount lustre-OST0000 complete [ 2821.600617] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2821.605499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2821.659356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2822.640480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2822.641273] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2827.747361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2827.765955] Lustre: Skipped 2 previous similar messages [ 2830.886818] Lustre: server umount lustre-MDT0000 complete [ 2831.848306] LustreError: 91638:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2831.873209] LustreError: 91638:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 2835.029904] LustreError: 91621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786180216 with bad export cookie 16471820280809935040 [ 2835.040806] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2835.086844] LustreError: 91621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2835.456659] Lustre: server umount lustre-MDT0001 complete [ 2844.743601] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 2847.323945] Key type lgssc unregistered [ 2847.662569] LNet: 95181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2847.667625] LNetError: 95181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2848.823784] LNet: Removed LNI 192.168.203.133@tcp [ 2849.401142] Key type .llcrypt unregistered [ 2849.403485] Key type ._llcrypt unregistered [ 2873.081903] Key type ._llcrypt registered [ 2873.084774] Key type .llcrypt registered [ 2873.253977] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 2874.922193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2874.979835] alg: No test for adler32 (adler32-zlib) [ 2875.959536] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 2876.263985] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 2877.927282] Key type lgssc registered [ 2878.904971] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2886.320784] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2886.340291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2901.919393] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2907.040213] LustreError: 96572:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.133@tcp: failed processing log, type 4: rc = -110 [ 2933.856249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2940.877388] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2948.181794] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2954.439592] Lustre: server umount lustre-OST0000 complete [ 2964.233150] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 05:12:23 (1786180343) [ 2970.831320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2971.112970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2975.549579] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2976.238738] LustreError: 98210:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2976.261224] LustreError: 98210:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 2981.347478] LustreError: 98209:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2983.650423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2983.926211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2987.373769] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2995.345755] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3001.002901] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3007.263648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3007.528353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3013.545222] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3015.776416] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3019.664362] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3031.011262] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3031.013373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3031.029975] Lustre: Skipped 1 previous similar message [ 3036.046703] Lustre: server umount lustre-OST0000 complete [ 3040.224238] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3040.229139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3040.242208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3040.246312] Lustre: Skipped 1 previous similar message [ 3041.250971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3041.260558] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3041.283815] Lustre: Skipped 1 previous similar message [ 3045.227831] Lustre: server umount lustre-MDT0000 complete [ 3046.412333] LustreError: 98210:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3046.433099] LustreError: 98210:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 3048.923673] LustreError: 98189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786180430 with bad export cookie 7150320380665155975 [ 3048.926517] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3048.932741] LustreError: 98189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3049.179603] Lustre: server umount lustre-MDT0001 complete [ 3057.794737] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 3060.014095] Key type lgssc unregistered [ 3060.290576] LNet: 101906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3060.298820] LNetError: 101906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3061.349746] LNet: Removed LNI 192.168.203.133@tcp [ 3061.904809] Key type .llcrypt unregistered [ 3061.908958] Key type ._llcrypt unregistered [ 3099.744560] Key type ._llcrypt registered [ 3099.748300] Key type .llcrypt registered [ 3099.880713] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 3101.173333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3101.236290] alg: No test for adler32 (adler32-zlib) [ 3102.419296] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 3102.619797] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 3104.335160] Key type lgssc registered [ 3105.698677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3113.478119] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3113.496347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3114.767976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3119.180530] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3125.217057] LustreError: 104090:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3125.236478] LustreError: 104090:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 3126.926969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3127.235237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3131.724744] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3138.171563] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3143.697330] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3151.506253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3151.697588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3157.647361] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3159.989731] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3164.496529] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3173.347526] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3173.369611] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3173.385881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3175.402414] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3175.406508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3175.419695] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3175.423182] Lustre: Skipped 1 previous similar message [ 3179.416943] Lustre: server umount lustre-OST0000 complete [ 3183.592245] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3183.609237] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3183.629552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3185.632908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3185.633186] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3188.844723] Lustre: server umount lustre-MDT0000 complete [ 3190.769379] LustreError: 106579:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3190.782877] LustreError: 106579:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 3192.286314] LustreError: 104069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786180573 with bad export cookie 16337578317516474455 [ 3192.290419] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3192.306510] LustreError: 104069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3192.532101] Lustre: server umount lustre-MDT0001 complete [ 3200.952102] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 3203.724217] Key type lgssc unregistered [ 3203.952856] LNet: 107626:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3203.962079] LNetError: 107626:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3203.979988] LNet: Removed LNI 192.168.203.133@tcp [ 3204.724906] Key type .llcrypt unregistered [ 3204.727298] Key type ._llcrypt unregistered [ 3218.849760] Key type ._llcrypt registered [ 3218.852191] Key type .llcrypt registered [ 3218.952453] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 3220.561039] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3220.610374] alg: No test for adler32 (adler32-zlib) [ 3221.603066] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 3221.795143] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 3223.423167] Key type lgssc registered [ 3224.656283] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3231.318587] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3231.342277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3246.943391] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3252.063447] LustreError: 108848:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.133@tcp: failed processing log, type 4: rc = -110 [ 3278.751188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3283.752857] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3289.362468] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3294.735475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3294.981455] LustreError: MGC192.168.203.133@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3294.990427] LustreError: 109873:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3294.995900] LustreError: 109873:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3295.001257] LustreError: 109873:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3295.013996] LustreError: 108855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786180676 with bad export cookie 15792787431106447941 [ 3295.032966] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3295.147482] Lustre: server umount lustre-MDT0000 complete [ 3295.154377] LustreError: 109873:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3321.984986] Lustre: server umount lustre-OST0000 complete [ 3330.302763] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 3339.068940] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 3367.915725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3368.071951] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3368.100832] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3368.171051] Lustre: lustre-MDT0000: new disk, initializing [ 3368.246134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3368.259247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3372.008194] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3381.182895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3381.369045] Lustre: 112668:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3381.433756] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3381.436958] Lustre: Skipped 1 previous similar message [ 3381.491236] Lustre: lustre-MDT0001: new disk, initializing [ 3381.567095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3381.599462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3381.616304] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3384.806772] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3393.734504] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3398.578251] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3404.324982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3404.485871] Lustre: lustre-OST0000: new disk, initializing [ 3404.488834] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3404.493364] Lustre: 114633:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3404.573105] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3409.601494] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3413.035515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3413.042211] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3413.120817] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3418.896746] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3424.148282] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3424.316834] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3428.797902] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3428.984840] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3433.470085] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3433.474120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3433.484819] Lustre: Skipped 1 previous similar message [ 3436.731725] Lustre: server umount lustre-OST0000 complete [ 3443.168865] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3443.173410] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3443.189181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3443.196085] Lustre: Skipped 1 previous similar message [ 3446.656902] Lustre: server umount lustre-MDT0000 complete [ 3450.181646] LustreError: 112658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786180831 with bad export cookie 15792787431106448046 [ 3450.195540] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3450.203039] LustreError: 112658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3450.473366] Lustre: server umount lustre-MDT0001 complete [ 3457.998775] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 05:20:38 (1786180838) [ 3460.980598] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3474.776451] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 3482.818413] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 3515.458677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3515.638583] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3515.666471] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3515.715360] Lustre: lustre-MDT0000: new disk, initializing [ 3515.770084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3515.782477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3520.060170] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3529.845598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3529.933184] Lustre: 119531:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3530.076530] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3530.093278] Lustre: Skipped 1 previous similar message [ 3530.354110] Lustre: lustre-MDT0001: new disk, initializing [ 3530.503369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3530.564849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3530.583786] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3535.216226] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3543.931541] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3550.064862] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3556.607187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3556.803106] Lustre: lustre-OST0000: new disk, initializing [ 3556.806915] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3556.814202] Lustre: 121498:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3556.889042] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3563.628332] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3566.098580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3566.109330] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3566.250995] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3573.872845] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3578.195525] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3578.458814] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3582.977223] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3583.192404] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3586.531991] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3586.535044] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3586.558715] Lustre: Skipped 1 previous similar message [ 3586.568133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3586.641904] Lustre: Skipped 2 previous similar messages [ 3591.649974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3591.660313] Lustre: Skipped 1 previous similar message [ 3596.779179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3596.782956] Lustre: Skipped 1 previous similar message [ 3599.327113] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3599.837324] Lustre: server umount lustre-OST0000 complete [ 3607.009287] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3607.016923] Lustre: Skipped 1 previous similar message [ 3607.033506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3607.044720] Lustre: Skipped 1 previous similar message [ 3609.861875] Lustre: server umount lustre-MDT0000 complete [ 3612.128519] LustreError: 119540:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3612.142500] LustreError: 119540:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 4 previous similar messages [ 3613.610393] LustreError: 120335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786180995 with bad export cookie 15792787431106448928 [ 3613.628358] LustreError: 120335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3613.632165] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3614.035620] Lustre: server umount lustre-MDT0001 complete [ 3621.355199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3621.784663] LustreError: 123725:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3621.951759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3626.319703] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3626.980080] LustreError: 123726:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3632.097499] LustreError: 123725:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3635.736254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3640.887926] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3649.069756] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3654.952533] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3660.904815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3666.866409] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3674.166793] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3683.868194] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3687.904371] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3687.909290] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3687.915020] Lustre: Skipped 1 previous similar message [ 3687.918385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3691.017133] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3694.391778] Lustre: server umount lustre-OST0000 complete [ 3701.219543] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3701.242105] Lustre: Skipped 2 previous similar messages [ 3704.229537] Lustre: server umount lustre-MDT0000 complete [ 3706.346772] LustreError: 123721:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3706.366489] LustreError: 123721:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 3708.300252] LustreError: 123707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786181089 with bad export cookie 15792787431106449810 [ 3708.321906] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3708.327643] LustreError: 123707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3708.852197] Lustre: server umount lustre-MDT0001 complete [ 3719.830210] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 3722.896951] Key type lgssc unregistered [ 3723.273944] LNet: 127358:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3723.279616] LNetError: 127358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3723.294166] LNet: Removed LNI 192.168.203.133@tcp [ 3724.324173] Key type .llcrypt unregistered [ 3724.334970] Key type ._llcrypt unregistered [ 3741.523131] Key type ._llcrypt registered [ 3741.525539] Key type .llcrypt registered [ 3741.616352] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 3756.226146] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 3757.411603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3757.438189] alg: No test for adler32 (adler32-zlib) [ 3758.401078] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 3758.646341] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 3760.383171] Key type lgssc registered [ 3761.191936] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3793.195190] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3793.231110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3794.487243] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3794.509864] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3794.573722] Lustre: lustre-MDT0000: new disk, initializing [ 3794.643194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3794.659969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3799.195344] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3811.611170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3811.731294] Lustre: 130511:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3811.800557] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3811.811864] Lustre: Skipped 1 previous similar message [ 3811.983768] Lustre: lustre-MDT0001: new disk, initializing [ 3812.164298] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3812.273418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3812.326056] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3816.673383] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3825.227710] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3830.619809] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3836.146614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3836.403748] Lustre: lustre-OST0000: new disk, initializing [ 3836.407437] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3836.412944] Lustre: 132483:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3836.481374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3841.983604] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3842.799396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3842.844467] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3843.008071] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3850.671288] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3854.576465] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3854.791317] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3859.541857] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3859.765828] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3863.531337] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3863.540183] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3863.547661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3863.560593] Lustre: Skipped 1 previous similar message [ 3867.321523] Lustre: server umount lustre-OST0000 complete [ 3873.761054] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3873.761413] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3873.762040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3873.772753] Lustre: Skipped 2 previous similar messages [ 3873.781837] LustreError: Skipped 1 previous similar message [ 3876.765076] Lustre: server umount lustre-MDT0000 complete [ 3879.944969] LustreError: 131310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786181261 with bad export cookie 12420041811538238115 [ 3879.945780] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3880.015719] LustreError: 131310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3880.320501] Lustre: server umount lustre-MDT0001 complete [ 3888.586718] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 05:27:48 (1786181268) [ 3894.460451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3894.799775] LustreError: 134896:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3894.813321] LustreError: 134896:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 3 previous similar messages [ 3894.858646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3899.109181] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3899.876485] LustreError: 134897:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3904.993457] LustreError: 134896:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3906.816555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3907.102555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3911.088228] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3917.919251] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3924.732832] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3927.522528] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3927.534763] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3927.550681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3927.554603] Lustre: Skipped 1 previous similar message [ 3930.592721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3930.594405] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3932.597224] Lustre: server umount lustre-MDT0000 complete [ 3936.657028] LustreError: 134876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786181318 with bad export cookie 12420041811538238997 [ 3936.675333] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3936.978117] Lustre: server umount lustre-MDT0001 complete [ 3945.370281] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 05:28:45 (1786181325) [ 3950.743815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3966.275339] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3971.423974] LustreError: 137308:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.133@tcp: failed processing log, type 4: rc = -110 [ 3997.215425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4003.317370] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4011.110447] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4013.398271] Lustre: server umount lustre-OST0000 complete [ 4022.864538] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 05:30:02 (1786181402) [ 4030.650271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4030.983036] LustreError: 138757:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4031.048471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4035.676310] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4036.080176] LustreError: 138757:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4036.109746] LustreError: 138757:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 4045.213029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4049.687600] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4056.973302] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4062.542600] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4068.707523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4068.928851] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4068.941423] Lustre: Skipped 1 previous similar message [ 4075.032329] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4082.071644] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4091.875277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4091.885767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4091.899527] Lustre: Skipped 2 previous similar messages [ 4091.928317] Lustre: Skipped 5 previous similar messages [ 4094.566249] Lustre: server umount lustre-MDT0000 complete [ 4096.996818] LustreError: 138752:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4097.015386] LustreError: 138752:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 3 previous similar messages [ 4097.627514] LustreError: 138737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786181479 with bad export cookie 12420041811538239655 [ 4097.637952] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4097.639503] LustreError: 138737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4097.946335] Lustre: server umount lustre-MDT0001 complete [ 4111.293717] Lustre: server umount lustre-OST0000 complete [ 4120.311947] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 05:31:40 (1786181500) [ 4125.897201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4126.176769] LustreError: 142410:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4130.092453] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4139.093905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4139.405389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4139.416035] Lustre: Skipped 1 previous similar message [ 4143.952945] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4152.268585] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4158.350855] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4165.391773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4170.993820] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4173.219897] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4180.768273] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4186.572759] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4187.027916] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4191.665947] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4191.844723] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4194.786905] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4194.801602] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4194.802152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4194.814667] Lustre: Skipped 1 previous similar message [ 4199.637489] Lustre: server umount lustre-OST0000 complete [ 4215.264916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4215.282776] Lustre: Skipped 5 previous similar messages [ 4217.311774] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4217.701402] Lustre: server umount lustre-MDT0000 complete [ 4220.389610] LustreError: 143137:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4220.407104] LustreError: 143137:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 3 previous similar messages [ 4222.219038] LustreError: 142392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786181603 with bad export cookie 12420041811538240733 [ 4222.226826] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4222.230132] LustreError: 142392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4222.489479] Lustre: server umount lustre-MDT0001 complete [ 4231.366611] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 05:33:31 (1786181611) [ 4237.453646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4253.087859] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4258.207395] LustreError: 146568:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.133@tcp: failed processing log, type 4: rc = -110 [ 4284.063423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4284.067660] Lustre: Skipped 1 previous similar message [ 4289.230475] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4297.214283] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4304.787300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4305.284948] LustreError: 146593:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4305.449201] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4310.299316] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4317.355151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4322.541773] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4329.383883] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4337.017269] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4341.584317] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4341.755627] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4346.026183] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4346.181668] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4348.384436] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4348.388705] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4348.395281] Lustre: Skipped 3 previous similar messages [ 4348.399880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4354.210754] Lustre: server umount lustre-MDT0000 complete [ 4358.134590] LustreError: 146576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786181739 with bad export cookie 12420041811538241755 [ 4358.153540] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4358.465721] Lustre: server umount lustre-MDT0001 complete [ 4373.244677] Lustre: server umount lustre-OST0000 complete [ 4382.352542] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 05:36:02 (1786181762) [ 4389.513273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4405.151807] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4410.274875] LustreError: 150764:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.133@tcp: failed processing log, type 4: rc = -110 [ 4441.099449] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4446.581667] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4451.536116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4451.816766] LustreError: 150789:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4451.831538] LustreError: 150789:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 5 previous similar messages [ 4451.942946] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4455.362850] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4462.510434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4466.755730] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4475.175401] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4481.030048] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4486.888493] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4487.045251] Lustre: lustre-OST0001: new disk, initializing [ 4487.048344] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4487.054060] Lustre: 153635:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4492.963929] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4493.604087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4493.625647] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4493.738137] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4501.145187] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4506.207984] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4506.352677] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4510.561468] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4510.770968] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4513.250773] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4513.262211] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4513.277528] Lustre: Skipped 2 previous similar messages [ 4513.280420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4513.283620] Lustre: Skipped 3 previous similar messages [ 4518.742450] Lustre: server umount lustre-OST0000 complete [ 4528.281286] Lustre: server umount lustre-OST0001 complete [ 4534.242038] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4534.253467] LustreError: Skipped 3 previous similar messages [ 4538.258622] Lustre: server umount lustre-MDT0000 complete [ 4541.616021] LustreError: 153634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786181923 with bad export cookie 12420041811538242707 [ 4541.620860] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4541.629721] LustreError: 153634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4542.141993] Lustre: server umount lustre-MDT0001 complete [ 4565.272716] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 05:39:05 (1786181945) [ 4566.865183] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4568.439770] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4570.641930] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 05:39:10 (1786181950) [ 4577.601917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4577.777320] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4577.796205] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4577.947687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4577.951924] Lustre: Skipped 6 previous similar messages [ 4582.150643] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4589.272939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4589.377565] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4589.402222] Lustre: 157195:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4593.471776] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4600.216181] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4605.928155] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4612.014360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4612.128566] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4618.330221] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4618.897892] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4625.870306] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4630.330777] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4630.557869] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4634.869642] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4635.193268] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4639.201428] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4639.217360] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4639.237135] Lustre: Skipped 5 previous similar messages [ 4643.151653] Lustre: server umount lustre-OST0000 complete [ 4664.531865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4671.131814] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4678.134704] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4680.197311] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4680.233030] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4680.234403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4683.492371] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4684.027200] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4688.312603] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4688.477612] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4693.497891] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9c5350f3d800.ost_server_uuid 50 [ 4695.439798] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9c5350f3d800.ost_server_uuid in FULL state after 0 sec [ 4695.712609] LustreError: 157203:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.33@tcp) returned error from blocking AST (req@0000000070c7ee4e x1872946675313024 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000a1be0d08/0xac5cda39303c75ed lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.33@tcp remote: 0x7c15de2cbd344bbf expref: 5 pid: 157976 timeout: 4795 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4695.744441] LustreError: lustre-MDT0000: A client on nid 192.168.203.33@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4695.752054] LustreError: 157191:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.33@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000a1be0d08/0xac5cda39303c75ed lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.33@tcp remote: 0x7c15de2cbd344bbf expref: 6 pid: 157976 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4700.655769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4700.661442] Lustre: Skipped 9 previous similar messages [ 4705.219706] Lustre: server umount lustre-OST0000 complete [ 4714.699717] Lustre: server umount lustre-MDT0000 complete [ 4716.003350] LustreError: 157201:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4716.048867] LustreError: 157201:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 9 previous similar messages [ 4717.777452] LustreError: 157186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786182099 with bad export cookie 12420041811538243890 [ 4717.792173] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4717.855216] LustreError: 157186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4732.383193] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4732.581425] Lustre: server umount lustre-MDT0001 complete [ 4743.545808] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 4746.064723] Key type lgssc unregistered [ 4746.348814] LNet: 163310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4746.353951] LNetError: 163310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4747.365243] LNet: Removed LNI 192.168.203.133@tcp [ 4748.146191] Key type .llcrypt unregistered [ 4748.149474] Key type ._llcrypt unregistered [ 4772.136301] Key type ._llcrypt registered [ 4772.138793] Key type .llcrypt registered [ 4772.377650] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 4774.207916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4774.273785] alg: No test for adler32 (adler32-zlib) [ 4775.287806] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 4775.464958] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 4777.135624] Key type lgssc registered [ 4778.414175] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4785.422883] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4785.443177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4786.890202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4791.080823] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4797.412866] LustreError: 164723:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4797.427505] LustreError: 164723:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 4798.339921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4798.599364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4802.793789] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4809.716583] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4815.892064] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4825.005569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4825.295803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4832.558527] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4833.490636] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4841.224633] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4848.248031] Lustre: Failing over lustre-MDT0000 [ 4848.416278] Lustre: server umount lustre-MDT0000 complete [ 4849.380034] LustreError: 164720:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4849.639076] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4849.652113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4853.217369] LustreError: 165450:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4853.219660] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4853.229454] LustreError: 165450:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 4853.240247] Lustre: Skipped 1 previous similar message [ 4858.338629] LustreError: 165780:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4858.348278] LustreError: 165780:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 3 previous similar messages [ 4860.556102] LustreError: 164705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.33@tcp arrived at 1786182242 with bad export cookie 6534615360297416276 [ 4860.584209] LustreError: 164705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4863.468342] LustreError: 167216:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4863.487736] LustreError: 167216:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 4866.944617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4867.020733] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4867.209179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4867.275974] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4871.416430] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4872.682860] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4872.704958] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4873.378592] Lustre: lustre-MDT0000: Denying connection for new client 4c1507e3-6349-4edf-a90f-87f02916870c (at 192.168.203.33@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:09 [ 4882.916028] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4882.929763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4882.931074] Lustre: Skipped 1 previous similar message [ 4882.939491] Lustre: Skipped 1 previous similar message [ 4887.359805] Lustre: server umount lustre-OST0000 complete [ 4891.490249] LustreError: 168451:0:(ldlm_lib.c:3002:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4891.496909] Lustre: 167801:0:(ldlm_lib.c:2402:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4891.504810] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 4891.677649] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 4891.689993] LustreError: 167801:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000fa5f1283 x1872947740454272/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4891.710292] LustreError: 167801:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 4891.715799] LustreError: 167801:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 4891.723826] Lustre: lustre-MDT0000: Recovery over after 0:19, of 2 clients 0 recovered and 2 were evicted. [ 4891.918848] Lustre: server umount lustre-MDT0000 complete [ 4893.152383] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4893.164236] LustreError: 164719:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4895.548701] LustreError: 167421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786182277 with bad export cookie 6534615360297416374 [ 4895.550390] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4895.570471] LustreError: 164033:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 4895.666569] LustreError: 167421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4898.105781] Lustre: server umount lustre-MDT0001 complete [ 4910.976146] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 4913.203979] Key type lgssc unregistered [ 4913.432763] LNet: 169205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4913.437260] LNetError: 169205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4913.450983] LNet: Removed LNI 192.168.203.133@tcp [ 4914.418788] Key type .llcrypt unregistered [ 4914.420453] Key type ._llcrypt unregistered [ 4938.778822] Key type ._llcrypt registered [ 4938.781418] Key type .llcrypt registered [ 4938.884262] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 4941.114721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4941.146397] alg: No test for adler32 (adler32-zlib) [ 4942.309214] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 4942.612506] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 4944.327150] Key type lgssc registered [ 4945.182665] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4953.319664] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4953.335845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4954.654214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4958.752829] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4964.834902] LustreError: 170619:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4964.890688] LustreError: 170619:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 4965.942765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4966.426596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4971.350854] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4978.748201] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4984.879554] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4991.173952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4991.351753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4996.705203] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5000.597345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5004.315856] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5010.913586] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5010.937548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5012.005204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5015.168891] Lustre: server umount lustre-OST0000 complete [ 5022.269270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5022.297028] Lustre: Skipped 1 previous similar message [ 5022.300956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5022.322424] Lustre: Skipped 2 previous similar messages [ 5024.938493] Lustre: server umount lustre-MDT0000 complete [ 5027.300202] LustreError: 170620:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5027.324796] LustreError: 170620:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 5029.203441] LustreError: 170599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786182410 with bad export cookie 18266060274485959879 [ 5029.211489] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5029.216032] LustreError: 170599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5029.582923] Lustre: server umount lustre-MDT0001 complete [ 5040.788669] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 5043.654188] Key type lgssc unregistered [ 5043.850852] LNet: 174154:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5043.862666] LNetError: 174154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5043.896856] LNet: Removed LNI 192.168.203.133@tcp [ 5044.616125] Key type .llcrypt unregistered [ 5044.617947] Key type ._llcrypt unregistered [ 5077.536483] Key type ._llcrypt registered [ 5077.541467] Key type .llcrypt registered [ 5077.740444] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 5079.236118] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5079.292430] alg: No test for adler32 (adler32-zlib) [ 5080.297250] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 5080.732476] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 5082.455136] Key type lgssc registered [ 5083.524718] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5090.643622] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5090.690769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5092.370976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5096.981560] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5101.537656] LustreError: 175837:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5101.549908] LustreError: 175837:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 5104.419251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5104.645354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5108.464741] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5115.586871] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5121.526437] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5128.476681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5128.844046] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5134.332248] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5137.087372] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5141.126401] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5147.620165] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5147.625494] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5147.633650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5147.640326] Lustre: Skipped 1 previous similar message [ 5149.372353] Lustre: server umount lustre-OST0000 complete [ 5152.738778] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5152.747016] Lustre: Skipped 1 previous similar message [ 5152.750400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5152.758047] Lustre: Skipped 1 previous similar message [ 5157.857666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5157.867205] Lustre: Skipped 1 previous similar message [ 5159.033749] Lustre: server umount lustre-MDT0000 complete [ 5162.424581] LustreError: 175817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786182543 with bad export cookie 4771804708824214350 [ 5162.443968] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5162.446464] LustreError: 175817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5162.687935] Lustre: server umount lustre-MDT0001 complete [ 5172.172468] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 5174.640119] Key type lgssc unregistered [ 5174.971100] LNet: 179373:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5174.989121] LNetError: 179373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5175.008733] LNet: Removed LNI 192.168.203.133@tcp [ 5175.873393] Key type .llcrypt unregistered [ 5175.878226] Key type ._llcrypt unregistered [ 5199.718720] Key type ._llcrypt registered [ 5199.729784] Key type .llcrypt registered [ 5199.930548] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 5201.434705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5201.511801] alg: No test for adler32 (adler32-zlib) [ 5202.837436] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 5203.046064] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 5204.735156] Key type lgssc registered [ 5205.626268] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5214.952949] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5214.963917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5216.411691] Lustre: *** cfs_fail_loc=135, val=0*** [ 5216.462946] LustreError: 180910:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5216.469127] LustreError: 180910:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.133@tcp: cfg command failed: rc = -2 [ 5216.478507] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5216.478507] [ 5216.490692] LustreError: MGC192.168.203.133@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5216.504820] LustreError: 180871:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5216.513532] LustreError: 180871:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5216.521813] LustreError: 180871:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5216.531966] LustreError: 180871:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5216.624444] Lustre: server umount lustre-MDT0000 complete [ 5216.627887] LustreError: 180871:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5223.083353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5223.462509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5227.404439] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5232.675566] LustreError: 181282:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5232.693516] LustreError: 181282:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 5235.150906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5235.467273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5239.611271] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5246.402625] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5250.873450] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5253.090673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5253.098614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5253.116546] Lustre: Skipped 1 previous similar message [ 5253.123935] Lustre: Skipped 1 previous similar message [ 5258.214291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5258.224019] Lustre: Skipped 1 previous similar message [ 5258.740296] Lustre: server umount lustre-MDT0000 complete [ 5260.259684] LustreError: 182336:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5260.274061] LustreError: 182336:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 5261.952961] LustreError: 181263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786182643 with bad export cookie 15874995786926490125 [ 5261.954255] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5261.962130] LustreError: 181263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5262.200696] Lustre: server umount lustre-MDT0001 complete [ 5273.373434] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 5276.176535] Key type lgssc unregistered [ 5276.398301] LNet: 183608:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5276.416858] LNetError: 183608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5276.433938] LNet: Removed LNI 192.168.203.133@tcp [ 5277.276799] Key type .llcrypt unregistered [ 5277.279630] Key type ._llcrypt unregistered [ 5296.652956] Key type ._llcrypt registered [ 5296.655793] Key type .llcrypt registered [ 5296.744946] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 5296.968492] Key type .llcrypt unregistered [ 5296.972299] Key type ._llcrypt unregistered [ 5315.497791] Key type ._llcrypt registered [ 5315.510771] Key type .llcrypt registered [ 5315.645973] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 5317.057274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5317.157164] alg: No test for adler32 (adler32-zlib) [ 5318.262615] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 5318.466952] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 5320.191537] Key type lgssc registered [ 5321.390463] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5330.126929] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5330.139972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5345.631724] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5350.751886] LustreError: 185743:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.133@tcp: failed processing log, type 4: rc = -110 [ 5377.376680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5383.244873] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5390.576543] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5397.268327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5397.353066] LustreError: 185078:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004c45af0b x1872948309788928/t0(0) o250->MGC192.168.203.133@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5397.629772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5397.679829] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5401.464846] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5403.108925] LustreError: 185768:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5403.154713] LustreError: 185768:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 5408.229158] LustreError: 185769:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5408.698818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5408.980584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5412.932659] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5419.899775] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5425.392986] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5428.122607] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5443.050725] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5443.080179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5443.092206] Lustre: Skipped 1 previous similar message [ 5448.162692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5448.174381] Lustre: Skipped 1 previous similar message [ 5453.283309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5453.290993] Lustre: Skipped 1 previous similar message [ 5455.327122] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5455.398501] Lustre: server umount lustre-OST0000 complete [ 5460.960474] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5460.963938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5460.970041] Lustre: Skipped 1 previous similar message [ 5460.973243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5463.528140] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5465.470096] Lustre: server umount lustre-MDT0000 complete [ 5468.642495] LustreError: 186783:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5468.660499] LustreError: 186783:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 5469.357806] LustreError: 185749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786182850 with bad export cookie 14199407314093007110 [ 5469.365275] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5469.389049] LustreError: 185749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5469.719546] Lustre: server umount lustre-MDT0001 complete [ 5483.070199] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 5485.437538] Key type lgssc unregistered [ 5485.733706] LNet: 189760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5485.751164] LNetError: 189760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5485.774558] LNet: Removed LNI 192.168.203.133@tcp [ 5486.759788] Key type .llcrypt unregistered [ 5486.761631] Key type ._llcrypt unregistered [ 5511.132954] Key type ._llcrypt registered [ 5511.137101] Key type .llcrypt registered [ 5511.210474] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 5512.398441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5512.521756] alg: No test for adler32 (adler32-zlib) [ 5514.138936] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 5514.473651] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 5516.207168] Key type lgssc registered [ 5516.929769] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5525.286295] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5525.311087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5526.748579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5530.340377] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5537.254588] LustreError: 191183:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5537.293393] LustreError: 191183:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 5539.989841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5540.247809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5544.554715] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5553.708383] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5559.632546] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5566.125466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5566.591209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5571.691097] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5572.831462] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5580.813817] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5587.555745] Lustre: Failing over lustre-MDT0000 [ 5587.723798] Lustre: server umount lustre-MDT0000 complete [ 5588.779910] LustreError: 191175:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5590.501546] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5590.518981] LustreError: 191179:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5590.535626] Lustre: Skipped 1 previous similar message [ 5593.791274] LustreError: 191174:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5593.829801] LustreError: 191174:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 5598.903285] LustreError: 191179:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5598.912855] LustreError: 191179:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 3 previous similar messages [ 5605.857745] Lustre: 190493:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786182971/real 1786182971] req@00000000f3fd4e1b x1872948515334016/t0(0) o400->MGC192.168.203.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786182987 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5606.199520] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5608.466261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5609.082981] LustreError: 192424:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5609.118768] LustreError: 192424:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 11 previous similar messages [ 5615.584418] LustreError: 190491:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000093bcf5f2 x1872948515339008/t0(0) o250->MGC192.168.203.133@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5615.800607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5615.843748] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5619.533971] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5620.450330] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5621.226329] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5621.242939] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5621.276668] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5623.876190] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5632.483906] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5641.699754] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5641.701271] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5641.713535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5641.732175] Lustre: Skipped 2 previous similar messages [ 5641.768861] Lustre: Skipped 1 previous similar message [ 5642.679455] Lustre: server umount lustre-OST0000 complete [ 5646.850928] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5646.851302] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5646.902218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5646.938833] Lustre: Skipped 1 previous similar message [ 5651.941196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5651.944209] Lustre: Skipped 1 previous similar message [ 5652.886044] Lustre: server umount lustre-MDT0000 complete [ 5656.971866] LustreError: 193029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786183038 with bad export cookie 6784112060667893821 [ 5656.991059] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5657.052030] LustreError: 193029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5657.410864] Lustre: server umount lustre-MDT0001 complete [ 5668.669353] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 5671.396484] Key type lgssc unregistered [ 5671.701748] LNet: 196119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5671.710894] LNetError: 196119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5671.719775] LNet: Removed LNI 192.168.203.133@tcp [ 5672.455217] Key type .llcrypt unregistered [ 5672.458215] Key type ._llcrypt unregistered [ 5698.419681] Key type ._llcrypt registered [ 5698.421899] Key type .llcrypt registered [ 5698.512177] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 5700.138921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5700.203297] alg: No test for adler32 (adler32-zlib) [ 5701.200977] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 5701.407353] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 5703.095177] Key type lgssc registered [ 5704.114605] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5710.867110] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5710.888970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5712.206099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5716.528255] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5721.709927] LustreError: 197534:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5721.716566] LustreError: 197534:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 5727.011555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5727.202039] LustreError: 197535:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5727.613767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5732.312448] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5739.821490] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5745.739590] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5752.278256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5752.546841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5756.681242] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5758.640749] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5766.097871] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5770.528952] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5778.152339] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5794.122889] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5802.995208] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5803.003663] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5803.008037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5803.015879] Lustre: Skipped 1 previous similar message [ 5803.024234] Lustre: Skipped 1 previous similar message [ 5807.096413] Lustre: server umount lustre-OST0000 complete [ 5813.223599] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5813.224570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5813.228083] Lustre: Skipped 1 previous similar message [ 5813.234195] Lustre: Skipped 1 previous similar message [ 5817.891288] Lustre: server umount lustre-MDT0000 complete [ 5818.339250] LustreError: 197531:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5818.356274] LustreError: 197531:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 5821.392594] LustreError: 200344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786183202 with bad export cookie 5992396305725978105 [ 5821.415100] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5821.415390] LustreError: 200344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5821.703187] Lustre: server umount lustre-MDT0001 complete [ 5833.586737] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 5835.747120] Key type lgssc unregistered [ 5835.968833] LNet: 201404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5835.975980] LNetError: 201404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5837.032431] LNet: Removed LNI 192.168.203.133@tcp [ 5837.895102] Key type .llcrypt unregistered [ 5837.899495] Key type ._llcrypt unregistered [ 5858.211173] Key type ._llcrypt registered [ 5858.213226] Key type .llcrypt registered [ 5858.360537] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 5859.382629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5859.399246] alg: No test for adler32 (adler32-zlib) [ 5860.792629] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 5861.027671] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 5862.695157] Key type lgssc registered [ 5863.754899] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5871.447436] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5871.460365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5872.822470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5878.465577] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5883.361163] LustreError: 202672:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5883.373389] LustreError: 202672:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 5886.313906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5886.661849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5890.822693] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5898.049976] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5904.032329] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5909.605399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5909.923419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5917.037811] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5918.393604] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 5924.293457] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5930.234624] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 5945.212429] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5956.201444] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 5962.271864] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 5974.497082] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5974.529873] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5974.551779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5975.011663] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5979.688546] Lustre: server umount lustre-OST0000 complete [ 5985.251128] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5985.252932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5985.258181] Lustre: Skipped 2 previous similar messages [ 5985.261994] Lustre: Skipped 2 previous similar messages [ 5985.264043] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5990.006748] Lustre: server umount lustre-MDT0000 complete [ 5990.371362] LustreError: 203411:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5990.384270] LustreError: 203411:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 5993.580732] LustreError: 206455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786183375 with bad export cookie 16723698622600003587 [ 5993.583354] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5993.591042] LustreError: 206455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5994.034459] Lustre: server umount lustre-MDT0001 complete [ 6006.956855] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 6009.047960] Key type lgssc unregistered [ 6009.294750] LNet: 207710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6009.306579] LNetError: 207710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6009.319896] LNet: Removed LNI 192.168.203.133@tcp [ 6010.243110] Key type .llcrypt unregistered [ 6010.246495] Key type ._llcrypt unregistered [ 6036.894503] Key type ._llcrypt registered [ 6036.897338] Key type .llcrypt registered [ 6036.989735] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 6038.591781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6038.691133] alg: No test for adler32 (adler32-zlib) [ 6040.044640] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 6040.239675] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 6041.903233] Key type lgssc registered [ 6042.872656] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6049.626984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6049.647224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6051.142755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6055.668511] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6061.540200] LustreError: 209125:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6061.562806] LustreError: 209125:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 6064.168414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6064.404229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6068.036745] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6077.096136] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6082.907767] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6090.386531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6090.764771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6096.819349] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6096.982303] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6104.526401] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6117.857583] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6117.869250] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6117.883366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6119.906351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6119.910081] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6119.918660] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6123.213638] Lustre: server umount lustre-OST0000 complete [ 6130.179787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6130.189562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6130.199322] Lustre: Skipped 1 previous similar message [ 6130.204376] Lustre: Skipped 1 previous similar message [ 6133.127625] Lustre: server umount lustre-MDT0000 complete [ 6135.265629] LustreError: 209125:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6135.277097] LustreError: 209125:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 6137.111250] LustreError: 209107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786183518 with bad export cookie 10743132620728764927 [ 6137.123441] LustreError: 209107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6137.148909] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6137.735433] Lustre: server umount lustre-MDT0001 complete [ 6148.939428] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 6152.013480] Key type lgssc unregistered [ 6152.418878] LNet: 212811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6152.424957] LNetError: 212811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6152.434775] LNet: Removed LNI 192.168.203.133@tcp [ 6153.232938] Key type .llcrypt unregistered [ 6153.235185] Key type ._llcrypt unregistered [ 6170.443389] Key type ._llcrypt registered [ 6170.445617] Key type .llcrypt registered [ 6170.526594] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 6171.957824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6171.974805] alg: No test for adler32 (adler32-zlib) [ 6173.080610] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 6173.255734] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 6174.975174] Key type lgssc registered [ 6176.011290] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6183.497787] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6183.537754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6184.868752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6188.955473] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6195.178954] LustreError: 214058:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6195.192963] LustreError: 214058:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 6197.624255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6197.921048] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6201.943394] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6208.198631] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6214.890570] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6222.226916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6222.562952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6228.724582] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6229.827338] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6235.609159] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6254.566802] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6254.581851] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6254.623139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6255.586870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6255.622798] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6255.648524] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6257.788882] Lustre: server umount lustre-OST0000 complete [ 6260.704903] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6260.717658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6260.721123] Lustre: Skipped 1 previous similar message [ 6265.825171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6265.830554] Lustre: Skipped 1 previous similar message [ 6266.775538] Lustre: server umount lustre-MDT0000 complete [ 6269.680391] LustreError: 214038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786183651 with bad export cookie 7394645823758981709 [ 6269.695128] LustreError: 214038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6269.699369] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6269.976874] Lustre: server umount lustre-MDT0001 complete [ 6281.980161] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 6284.036441] Key type lgssc unregistered [ 6284.361969] LNet: 217747:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6284.368300] LNetError: 217747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6284.529184] LNet: Removed LNI 192.168.203.133@tcp [ 6285.129241] Key type .llcrypt unregistered [ 6285.135790] Key type ._llcrypt unregistered [ 6306.159984] Key type ._llcrypt registered [ 6306.161910] Key type .llcrypt registered [ 6306.299911] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 6307.357972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6307.651366] alg: No test for adler32 (adler32-zlib) [ 6309.768904] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 6310.000955] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 6311.687971] Key type lgssc registered [ 6312.784267] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6319.948942] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6319.966917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6321.289901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6325.416228] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6330.858706] LustreError: 219012:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6330.877510] LustreError: 219012:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 6334.908612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6335.183188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6340.098490] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6347.477094] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6352.550454] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6359.608789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6359.890897] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6366.156968] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6367.698309] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6376.080813] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6387.261092] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6387.299933] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6387.486923] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6393.661246] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6401.627790] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6403.054124] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6415.482322] Lustre: Permanently deactivating lustre-OST0001 [ 6415.497850] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6421.008378] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6432.965517] Lustre: Permanently reactivating lustre-OST0001 [ 6432.975280] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6432.982861] Lustre: Skipped 2 previous similar messages [ 6441.479760] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6441.509782] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6442.907986] Lustre: server umount lustre-OST0001 complete [ 6446.608994] LustreError: 222271:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6448.103622] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6448.107487] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6448.119451] LustreError: Skipped 1 previous similar message [ 6448.121983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6448.136965] Lustre: Skipped 1 previous similar message [ 6448.160366] Lustre: Skipped 1 previous similar message [ 6451.688663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6451.691980] LustreError: 222271:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6451.697425] Lustre: Skipped 1 previous similar message [ 6453.208982] Lustre: server umount lustre-OST0000 complete [ 6458.336374] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6458.346419] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6458.365315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6461.919315] LustreError: 222918:0:(osp_dev.c:699:osp_process_config()) lustre-OST0001-osc-MDT0001: unknown param osc.active=1 [ 6461.932445] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6467.040807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6467.057210] Lustre: Skipped 3 previous similar messages [ 6468.132788] Lustre: server umount lustre-MDT0000 complete [ 6471.809811] LustreError: 218994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786183853 with bad export cookie 9679645303951566949 [ 6471.819663] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6471.875171] LustreError: 218994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6472.304226] Lustre: server umount lustre-MDT0001 complete [ 6483.182358] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 6485.314660] Key type lgssc unregistered [ 6485.773777] LNet: 224076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6485.778443] LNetError: 224076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6486.884367] LNet: Removed LNI 192.168.203.133@tcp [ 6487.724809] Key type .llcrypt unregistered [ 6487.726682] Key type ._llcrypt unregistered [ 6530.244849] Key type ._llcrypt registered [ 6530.246570] Key type .llcrypt registered [ 6530.360851] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 6533.021286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6533.134461] alg: No test for adler32 (adler32-zlib) [ 6534.541752] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 6534.729722] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 6536.427160] Key type lgssc registered [ 6538.016449] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6545.718273] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6545.754053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6547.116750] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6547.128526] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6547.246713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6551.457282] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6559.505292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6559.583098] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6559.608621] Lustre: 226280:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6559.816951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6564.205944] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6571.802443] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6581.422408] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6587.737890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6587.851575] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6588.101858] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6593.894618] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6597.750434] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6600.828223] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6605.444872] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6613.171976] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6620.028593] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6629.675699] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6640.009538] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6656.863513] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6656.869678] Lustre: Skipped 1 previous similar message [ 6688.901499] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6688.907804] Lustre: Skipped 3 previous similar messages [ 6755.667166] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6755.674908] Lustre: Skipped 8 previous similar messages [ 6782.439393] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6782.439891] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6782.450537] LustreError: Skipped 1 previous similar message [ 6782.452182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6782.477521] Lustre: Skipped 1 previous similar message [ 6784.299529] Lustre: server umount lustre-OST0000 complete [ 6790.114325] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6790.118579] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6790.131908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6790.138681] Lustre: Skipped 1 previous similar message [ 6792.674153] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6792.676418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6792.686151] Lustre: Skipped 1 previous similar message [ 6797.794067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6797.802932] Lustre: Skipped 1 previous similar message [ 6802.399141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6804.641666] Lustre: server umount lustre-MDT0000 complete [ 6808.049182] LustreError: 226291:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6808.119584] LustreError: 226291:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 4 previous similar messages [ 6808.646647] LustreError: 226274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786184190 with bad export cookie 16098950385381609654 [ 6808.649037] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6808.674331] LustreError: 226274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6808.987325] Lustre: server umount lustre-MDT0001 complete [ 6821.257729] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 6823.469478] Key type lgssc unregistered [ 6823.703802] LNet: 231063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6823.709854] LNetError: 231063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6824.741491] LNet: Removed LNI 192.168.203.133@tcp [ 6825.759416] Key type .llcrypt unregistered [ 6825.761581] Key type ._llcrypt unregistered [ 6851.642614] Key type ._llcrypt registered [ 6851.646549] Key type .llcrypt registered [ 6851.803242] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 6854.084878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6854.139919] alg: No test for adler32 (adler32-zlib) [ 6855.156333] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 6855.359391] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 6857.111154] Key type lgssc registered [ 6858.133740] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6865.319856] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6865.346204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6866.710470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6870.731898] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6877.154945] LustreError: 232480:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6877.170680] LustreError: 232480:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 6879.411698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6879.674766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6883.783715] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6890.322859] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6896.727284] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6901.966782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6902.240778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6908.681568] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6912.525192] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 6916.002772] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6935.663841] Lustre: server umount lustre-OST0000 complete [ 6941.154351] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6941.160104] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6941.169725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6943.200449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6943.203866] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6948.325266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6948.336642] Lustre: Skipped 1 previous similar message [ 6949.446647] Lustre: server umount lustre-MDT0000 complete [ 6952.415942] Lustre: 231792:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786184317/real 1786184317] req@000000001babbdee x1872949921484032/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786184333 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6952.459755] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6952.477694] LustreError: 232475:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6952.496142] LustreError: 232475:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 6952.920985] LustreError: 232459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786184334 with bad export cookie 14582527647303224163 [ 6952.927073] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6952.947629] LustreError: 232459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6953.175570] Lustre: server umount lustre-MDT0001 complete [ 6964.182986] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 6966.515810] Key type lgssc unregistered [ 6966.706601] LNet: 236171:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6966.712570] LNetError: 236171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6966.723910] LNet: Removed LNI 192.168.203.133@tcp [ 6967.309656] Key type .llcrypt unregistered [ 6967.312226] Key type ._llcrypt unregistered [ 6992.572516] Key type ._llcrypt registered [ 6992.574918] Key type .llcrypt registered [ 6992.649066] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 6994.550772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6994.799104] alg: No test for adler32 (adler32-zlib) [ 6996.159243] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 6996.327664] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 6997.959501] Key type lgssc registered [ 6999.231605] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7269.726705] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 7271.929562] Key type lgssc unregistered [ 7272.120849] LNet: 238198:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7272.125679] LNetError: 238198:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7272.140164] LNet: Removed LNI 192.168.203.133@tcp [ 7272.886390] Key type .llcrypt unregistered [ 7272.894731] Key type ._llcrypt unregistered [ 7296.793336] Key type ._llcrypt registered [ 7296.795694] Key type .llcrypt registered [ 7296.876939] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 7316.067618] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 7317.363229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7317.408474] alg: No test for adler32 (adler32-zlib) [ 7318.540626] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 7318.752157] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 7320.415161] Key type lgssc registered [ 7321.710993] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7352.789663] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7041 sec ========== 06:25:32 (1786184732) [ 7354.720896] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 06:25:34 (1786184734) === [ 7358.666644] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 06:25:38 (1786184738) === [ 7397.764416] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 7400.534862] Key type lgssc unregistered [ 7400.910288] LNet: 243437:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7400.919753] LNetError: 243437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7400.942756] LNet: Removed LNI 192.168.203.133@tcp [ 7401.651600] Key type .llcrypt unregistered [ 7401.656218] Key type ._llcrypt unregistered