[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 522354565 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.002403] x2apic enabled [ 0.003012] Switched APIC routing to physical x2apic. [ 0.004034] kvm-guest: setup PV IPIs [ 0.007436] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008032] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009020] pid_max: default: 32768 minimum: 301 [ 0.010159] LSM: Security Framework initializing [ 0.011077] Yama: becoming mindful. [ 0.012070] SELinux: Initializing. [ 0.013114] *** VALIDATE selinux *** [ 0.021732] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025834] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026197] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027134] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028155] *** VALIDATE tmpfs *** [ 0.030462] *** VALIDATE proc *** [ 0.032214] *** VALIDATE cgroup *** [ 0.033020] *** VALIDATE cgroup2 *** [ 0.034309] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035193] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036015] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037047] Spectre V2 : User space: Vulnerable [ 0.038016] Speculative Store Bypass: Vulnerable [ 0.041271] debug: unmapping init [mem 0xffffffffbe059000-0xffffffffbe060fff] [ 0.043210] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044806] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045034] ... version: 2 [ 0.046023] ... bit width: 48 [ 0.047020] ... generic registers: 4 [ 0.048021] ... value mask: 0000ffffffffffff [ 0.049021] ... max period: 00007fffffffffff [ 0.050021] ... fixed-purpose events: 3 [ 0.051017] ... event mask: 000000070000000f [ 0.052373] rcu: Hierarchical SRCU implementation. [ 0.054815] smp: Bringing up secondary CPUs ... [ 0.055683] x86: Booting SMP configuration: [ 0.056044] .... node #0, CPUs: #1 #2 #3 [ 0.059684] smp: Brought up 1 node, 4 CPUs [ 0.061020] smpboot: Max logical packages: 1 [ 0.062029] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.270039] node 0 deferred pages initialised in 206ms [ 0.275168] devtmpfs: initialized [ 0.276299] x86/mm: Memory block size: 128MB [ 0.280209] gcov: version magic: 0x41383552 [ 0.283382] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.284073] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.287375] pinctrl core: initialized pinctrl subsystem [ 0.289228] [ 0.291013] ************************************************************* [ 0.293016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.296016] ** ** [ 0.299014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.302017] ** ** [ 0.305015] ** This means that this kernel is built to expose internal ** [ 0.307015] ** IOMMU data structures, which may compromise security on ** [ 0.310014] ** your system. ** [ 0.313017] ** ** [ 0.315021] ** If you see this message and you are not debugging the ** [ 0.318013] ** kernel, report this immediately to your vendor! ** [ 0.321017] ** ** [ 0.325025] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.328016] ************************************************************* [ 0.332091] NET: Registered protocol family 16 [ 0.333453] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.338068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.341073] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.345158] cpuidle: using governor menu [ 0.347753] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.350542] PCI: Using configuration type 1 for base access [ 0.352141] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.362130] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.365070] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.369043] cryptd: max_cpu_qlen set to 1000 [ 0.372307] ACPI: Added _OSI(Module Device) [ 0.374022] ACPI: Added _OSI(Processor Device) [ 0.375017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.377025] ACPI: Added _OSI(Processor Aggregator Device) [ 0.383579] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.390718] ACPI: Interpreter enabled [ 0.392067] ACPI: PM: (supports S0 S3 S4 S5) [ 0.394030] ACPI: Using IOAPIC for interrupt routing [ 0.396113] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.401497] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.413441] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.416047] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.419022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.423101] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.429499] acpiphp: Slot [2] registered [ 0.431214] acpiphp: Slot [5] registered [ 0.433188] acpiphp: Slot [6] registered [ 0.435140] acpiphp: Slot [7] registered [ 0.436137] acpiphp: Slot [8] registered [ 0.438145] acpiphp: Slot [9] registered [ 0.440126] acpiphp: Slot [10] registered [ 0.441214] acpiphp: Slot [3] registered [ 0.444174] acpiphp: Slot [4] registered [ 0.445378] acpiphp: Slot [11] registered [ 0.448156] acpiphp: Slot [12] registered [ 0.450205] acpiphp: Slot [13] registered [ 0.451194] acpiphp: Slot [14] registered [ 0.453281] acpiphp: Slot [15] registered [ 0.456215] acpiphp: Slot [16] registered [ 0.458123] acpiphp: Slot [17] registered [ 0.459105] acpiphp: Slot [18] registered [ 0.461120] acpiphp: Slot [19] registered [ 0.462126] acpiphp: Slot [20] registered [ 0.464140] acpiphp: Slot [21] registered [ 0.466120] acpiphp: Slot [22] registered [ 0.467125] acpiphp: Slot [23] registered [ 0.469113] acpiphp: Slot [24] registered [ 0.471137] acpiphp: Slot [25] registered [ 0.473148] acpiphp: Slot [26] registered [ 0.474145] acpiphp: Slot [27] registered [ 0.476127] acpiphp: Slot [28] registered [ 0.478147] acpiphp: Slot [29] registered [ 0.480118] acpiphp: Slot [30] registered [ 0.481095] acpiphp: Slot [31] registered [ 0.483099] PCI host bridge to bus 0000:00 [ 0.485026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.488026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.491027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.494030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.496027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.499027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.502211] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.505178] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.508433] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.521020] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.528070] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.531020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.534023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.536018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.539606] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.542789] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.546057] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.550868] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.558015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.569021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.576030] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.582012] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.595018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.603017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.626022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.638974] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.646017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.654017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.676018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.688415] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.694000] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.701086] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.721019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.733024] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.739016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.747016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.773019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.785271] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.794017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.799015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.815015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.825238] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.830018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.835016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.851022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.863080] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.865368] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.868361] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.871419] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.873335] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.878122] iommu: Default domain type: Passthrough [ 0.880501] SCSI subsystem initialized [ 0.883187] ACPI: bus type USB registered [ 0.885110] usbcore: registered new interface driver usbfs [ 0.888104] usbcore: registered new interface driver hub [ 0.890078] usbcore: registered new device driver usb [ 0.892195] pps_core: LinuxPPS API ver. 1 registered [ 0.894012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.898062] PTP clock support registered [ 0.900114] EDAC MC: Ver: 3.0.0 [ 0.902100] PCI: Using ACPI for IRQ routing [ 0.904999] NetLabel: Initializing [ 0.906013] NetLabel: domain hash size = 128 [ 0.907008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.909082] NetLabel: unlabeled traffic allowed by default [ 0.912043] vgaarb: loaded [ 0.913210] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.914011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.921000] clocksource: Switched to clocksource kvm-clock [ 1.032384] VFS: Disk quotas dquot_6.6.0 [ 1.034639] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.038355] *** VALIDATE ramfs *** [ 1.039774] *** VALIDATE hugetlbfs *** [ 1.041570] pnp: PnP ACPI init [ 1.044236] pnp: PnP ACPI: found 6 devices [ 1.069537] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.073906] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.076678] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.079540] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.082646] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.085535] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.089129] NET: Registered protocol family 2 [ 1.092421] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.098624] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.103218] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.109438] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.113465] TCP: Hash tables configured (established 65536 bind 65536) [ 1.117172] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.121241] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.124920] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.128924] NET: Registered protocol family 1 [ 1.132332] RPC: Registered named UNIX socket transport module. [ 1.134544] RPC: Registered udp transport module. [ 1.136413] RPC: Registered tcp transport module. [ 1.138636] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.141290] NET: Registered protocol family 44 [ 1.143063] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.145250] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.147561] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.150178] PCI: CLS 0 bytes, default 64 [ 1.152262] Unpacking initramfs... [ 2.550052] debug: unmapping init [mem 0xffff9e4bbcc54000-0xffff9e4bbffbffff] [ 2.554300] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.556651] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.560193] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.062120] Initialise system trusted keyrings [ 3.063904] Key type blacklist registered [ 3.066866] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.079042] zbud: loaded [ 3.082668] *** VALIDATE nfs *** [ 3.084204] *** VALIDATE nfs4 *** [ 3.086028] pstore: using deflate compression [ 3.090276] Platform Keyring initialized [ 3.190367] NET: Registered protocol family 38 [ 3.191955] Key type asymmetric registered [ 3.193703] Asymmetric key parser 'x509' registered [ 3.195862] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.199347] io scheduler mq-deadline registered [ 3.200885] io scheduler kyber registered [ 3.202820] io scheduler bfq registered [ 3.205078] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.208251] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.212310] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.215674] ACPI: Power Button [PWRF] [ 3.222627] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.229387] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.261557] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.270947] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.301047] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.330525] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.359992] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.365223] Non-volatile memory driver v1.3 [ 3.367305] Linux agpgart interface v0.103 [ 3.402612] virtio_blk virtio1: [vda] 146624 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.407446] vda: detected capacity change from 0 to 75071488 [ 3.427888] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.432128] vdb: detected capacity change from 0 to 1073741824 [ 3.451287] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.456210] vdc: detected capacity change from 0 to 2621440000 [ 3.471519] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.474957] vdd: detected capacity change from 0 to 2621440000 [ 3.491217] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.494315] vde: detected capacity change from 0 to 4294967296 [ 3.512333] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.515626] vdf: detected capacity change from 0 to 4294967296 [ 3.524906] libphy: Fixed MDIO Bus: probed [ 3.531458] usbcore: registered new interface driver usbserial_generic [ 3.533854] usbserial: USB Serial support registered for generic [ 3.536979] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.542027] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.544182] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.546763] mousedev: PS/2 mouse device common for all mice [ 3.550280] rtc_cmos 00:05: RTC can wake from S4 [ 3.551438] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.557057] rtc_cmos 00:05: registered as rtc0 [ 3.565102] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.566380] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.571567] intel_pstate: CPU model not supported [ 3.574255] hid: raw HID events driver (C) Jiri Kosina [ 3.580877] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.583554] usbcore: registered new interface driver usbhid [ 3.583564] usbhid: USB HID core driver [ 3.583707] drop_monitor: Initializing network drop monitor service [ 3.583852] Initializing XFRM netlink socket [ 3.585349] NET: Registered protocol family 10 [ 3.601559] Segment Routing with IPv6 [ 3.603618] NET: Registered protocol family 17 [ 3.605619] mpls_gso: MPLS GSO support [ 3.613397] RAS: Correctable Errors collector initialized. [ 3.616130] AVX version of gcm_enc/dec engaged. [ 3.618125] AES CTR mode by8 optimization enabled [ 3.714529] sched_clock: Marking stable (3714490939, 0)->(4773859453, -1059368514) [ 3.718787] registered taskstats version 1 [ 3.721264] Loading compiled-in X.509 certificates [ 3.722860] zswap: loaded using pool lzo/zbud [ 3.750167] Key type big_key registered [ 3.760572] Key type encrypted registered [ 3.762541] ima: No TPM chip found, activating TPM-bypass! [ 3.765060] ima: Allocated hash algorithm: sha1 [ 3.767102] ima: No architecture policies found [ 3.769211] evm: Initialising EVM extended attributes: [ 3.771499] evm: security.selinux [ 3.773073] evm: security.ima [ 3.774357] evm: security.capability [ 3.776024] evm: HMAC attrs: 0x1 [ 3.778956] rtc_cmos 00:05: setting system clock to 2026-09-04 01:21:06 UTC (1788484866) [ 3.787492] debug: unmapping init [mem 0xffffffffbf003000-0xffffffffbf1fffff] [ 3.791020] debug: unmapping init [mem 0xffffffffbdd82000-0xffffffffbe058fff] [ 3.800152] Write protecting the kernel read-only data: 28672k [ 3.804001] debug: unmapping init [mem 0xffffffffbc403000-0xffffffffbc5fffff] [ 3.807216] debug: unmapping init [mem 0xffffffffbcd14000-0xffffffffbcdfffff] [ 3.840488] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.849412] systemd[1]: Detected virtualization kvm. [ 3.852284] systemd[1]: Detected architecture x86-64. [ 3.854928] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.880942] systemd[1]: No hostname configured. [ 3.883282] systemd[1]: Set hostname to . [ 3.886170] random: systemd: uninitialized urandom read (16 bytes read) [ 3.889555] systemd[1]: Initializing machine ID from random generator. [ 3.935040] random: ln: uninitialized urandom read (6 bytes read) [ 4.055190] random: systemd: uninitialized urandom read (16 bytes read) [ 4.058272] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.062902] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.067882] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Swap. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.679879] device-mapper: uevent: version 1.0.3 [ 4.682244] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... [ 5.210249] random: fast init done Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.352911] virtio_net virtio0 ens2: renamed from eth0 [ 5.440458] scsi host0: ata_piix [ 5.461863] scsi host1: ata_piix [ 5.463618] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.466262] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.729244] dracut-initqueue[595]: RTNETLINK answers: File exists [ 10.281421] random: crng init done [ 10.284748] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.618898] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.774133] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.056357] SELinux: Disabled at runtime. [ 12.117418] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.126510] systemd[1]: Detected virtualization kvm. [ 12.128612] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.612799] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.616606] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.625504] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.629624] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.633417] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.640231] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.643499] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting Kernel Debug File System... Mountin[ 12.725608] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS g Huge Pages File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.034244] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.347766] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.386396] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.479089] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.490867] EDAC sbridge: Ver: 1.1.2 [ 15.139499] Key type dns_resolver registered [ 15.435207] NFS: Registering the id_resolver key type [ 15.437183] Key type id_resolver registered [ 15.438546] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg337-server login: [ 41.488423] libcfs: loading out-of-tree module taints kernel. [ 41.509108] Key type ._llcrypt registered [ 41.510829] Key type .llcrypt registered [ 41.559226] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 59.904711] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 61.536729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 61.552432] alg: No test for adler32 (adler32-zlib) [ 62.902797] Lustre: Lustre: Build Version: 2.17.57_84_g5b8c505 [ 63.713718] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 65.455226] Key type lgssc registered [ 67.273789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 71.362083] hrtimer: interrupt took 2128185 ns [ 88.340865] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 130.451345] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 146.123512] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 146.191054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 147.576079] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 147.604163] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 147.710432] Lustre: lustre-MDT0000: new disk, initializing [ 147.818158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 147.830050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 152.145609] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 165.933682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 166.017912] Lustre: 6524:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 166.050377] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 166.056141] Lustre: Skipped 1 previous similar message [ 166.146621] Lustre: lustre-MDT0001: new disk, initializing [ 166.234578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 166.282977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 166.302443] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 171.228744] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 176.181423] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 187.020708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 187.545457] Lustre: lustre-OST0000: new disk, initializing [ 187.555537] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 187.563951] Lustre: 8463:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 187.643709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 189.519061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 189.537148] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 189.655767] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 194.408953] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 210.031858] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 210.237887] Lustre: lustre-OST0001: new disk, initializing [ 210.241277] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 210.247888] Lustre: 9536:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 210.331145] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 216.685592] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 218.679460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 218.693880] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 218.803820] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 228.251064] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 239.622910] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 246.710191] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing check_logdir /tmp/testlogs/ [ 252.751568] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing yml_node [ 258.382491] Lustre: DEBUG MARKER: Client: 2.17.57.84 [ 261.613600] Lustre: DEBUG MARKER: MDS: 2.17.57.84 [ 264.837221] Lustre: DEBUG MARKER: OSS: 2.17.57.84 [ 266.875390] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 21:25:26 EDT 2026 [ 283.796623] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 285.323516] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 304.611348] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 304.617121] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 304.630592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 306.143647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 306.145370] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 306.153760] Lustre: Skipped 1 previous similar message [ 306.163124] Lustre: Skipped 1 previous similar message [ 308.013058] Lustre: server umount lustre-MDT0000 complete [ 312.891804] LustreError: 6517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788485175 with bad export cookie 3077038769543522779 [ 312.894693] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 312.903465] LustreError: 6517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 316.396988] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 316.399159] LustreError: 6531:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 316.399308] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 316.414957] Lustre: Skipped 2 previous similar messages [ 316.430787] LustreError: 6531:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 317.410256] LustreError: 6530:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 317.444366] LustreError: 6530:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 319.559504] Lustre: server umount lustre-MDT0001 complete [ 324.697690] Lustre: server umount lustre-OST0000 complete [ 328.511659] Lustre: server umount lustre-OST0001 complete [ 335.536415] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 346.972927] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 383.664943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 383.896715] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 383.926397] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 384.009252] Lustre: lustre-MDT0000: new disk, initializing [ 384.091810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 384.113431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 388.381794] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 399.107063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 399.240551] Lustre: 15088:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 399.321269] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 399.325532] Lustre: Skipped 1 previous similar message [ 399.437921] Lustre: lustre-MDT0001: new disk, initializing [ 399.602254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 399.654867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 399.675590] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 404.482618] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 413.805336] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 418.871255] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 425.319489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 425.539393] Lustre: lustre-OST0000: new disk, initializing [ 425.543314] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 425.548923] Lustre: 17059:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 425.644251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 427.125469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 427.141190] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 427.249190] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 431.642731] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 440.946053] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 445.717428] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 445.896528] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 449.921211] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 450.106084] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 453.094693] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 453.095719] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 453.098887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 453.098895] Lustre: Skipped 3 previous similar messages [ 453.109052] LustreError: Skipped 1 previous similar message [ 453.148413] Lustre: Skipped 1 previous similar message [ 458.212797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 458.218124] Lustre: Skipped 2 previous similar messages [ 466.399130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 466.566208] Lustre: server umount lustre-OST0000 complete [ 471.520885] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 471.527717] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 471.536461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 471.539426] Lustre: Skipped 2 previous similar messages [ 484.320600] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 484.746934] Lustre: server umount lustre-MDT0000 complete [ 488.931515] LustreError: 15102:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 488.967632] LustreError: 15102:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 489.731536] LustreError: 15082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788485352 with bad export cookie 3077038769543524312 [ 489.742609] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 489.747139] LustreError: 15082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 490.311393] Lustre: server umount lustre-MDT0001 complete [ 493.726282] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 21:29:14 (1788485354) [ 499.979778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 500.409086] LustreError: 19325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 500.492934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 503.999618] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 505.826217] LustreError: 19326:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 511.339750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 516.286841] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 524.271409] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 530.082576] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 535.948257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 541.856757] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 547.467422] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 556.006365] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 556.016632] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 556.020624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 556.020633] Lustre: Skipped 6 previous similar messages [ 556.058483] Lustre: Skipped 2 previous similar messages [ 559.748032] Lustre: server umount lustre-OST0000 complete [ 569.986668] Lustre: server umount lustre-MDT0000 complete [ 571.360908] LustreError: 19325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 571.371714] LustreError: 19325:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 573.513717] LustreError: 19305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788485436 with bad export cookie 3077038769543525194 [ 573.520797] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 573.529188] LustreError: 19305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 573.784515] Lustre: server umount lustre-MDT0001 complete [ 583.305542] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 585.969911] Key type lgssc unregistered [ 586.253140] LNet: 22854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 586.266358] LNetError: 22854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 586.298451] LNet: Removed LNI 192.168.203.137@tcp [ 587.268134] Key type .llcrypt unregistered [ 587.275050] Key type ._llcrypt unregistered [ 608.837368] Key type ._llcrypt registered [ 608.839118] Key type .llcrypt registered [ 609.016391] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 610.337461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 610.365850] alg: No test for adler32 (adler32-zlib) [ 611.448557] Lustre: Lustre: Build Version: 2.17.57_84_g5b8c505 [ 611.632541] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 613.271163] Key type lgssc registered [ 614.175641] Lustre: Echo OBD driver; http://www.lustre.org/ [ 620.889409] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 620.906567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 622.248573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 626.463213] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 631.783550] LustreError: 24261:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 631.812506] LustreError: 24261:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 634.209340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 634.573409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 638.914942] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 645.667996] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 650.565926] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 656.468690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 656.753523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 662.025700] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 667.009105] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 668.111593] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 682.471540] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 682.484391] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 682.513845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 684.525984] Lustre: server umount lustre-OST0000 complete [ 691.170492] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 691.183318] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 691.193789] Lustre: Skipped 1 previous similar message [ 691.197920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 691.202222] Lustre: Skipped 1 previous similar message [ 692.704903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 692.710888] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 694.139100] Lustre: server umount lustre-MDT0000 complete [ 697.327522] LustreError: 27485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788485560 with bad export cookie 923258497667811979 [ 697.331517] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 697.340920] LustreError: 27485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 697.623404] Lustre: server umount lustre-MDT0001 complete [ 707.277865] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 709.975979] Key type lgssc unregistered [ 710.331628] LNet: 28234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 710.345586] LNetError: 28234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 710.370801] LNet: Removed LNI 192.168.203.137@tcp [ 711.052333] Key type .llcrypt unregistered [ 711.059192] Key type ._llcrypt unregistered [ 732.540656] Key type ._llcrypt registered [ 732.543649] Key type .llcrypt registered [ 732.635794] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 733.745782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 733.774684] alg: No test for adler32 (adler32-zlib) [ 734.897478] Lustre: Lustre: Build Version: 2.17.57_84_g5b8c505 [ 735.153396] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 736.816557] Key type lgssc registered [ 737.851731] Lustre: Echo OBD driver; http://www.lustre.org/ [ 744.825269] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 744.842753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 746.225518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 750.049227] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 755.685631] LustreError: 29644:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 755.701487] LustreError: 29644:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 757.460849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 757.787528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 761.979179] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 769.622037] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 775.353640] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 787.035756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 787.426440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 790.506209] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 793.504716] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 801.436657] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 807.914064] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 807.916739] LustreError: lustre-OST0000-osc-MDT0001: operation ost_create to node 0@lo failed: rc = -107 [ 807.918530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 807.940043] Lustre: Skipped 1 previous similar message [ 809.354442] Lustre: server umount lustre-OST0000 complete [ 814.048183] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 814.056896] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 814.072867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 814.078361] Lustre: Skipped 1 previous similar message [ 818.145060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 818.147905] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 819.436596] Lustre: server umount lustre-MDT0000 complete [ 822.585861] LustreError: 29625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788485685 with bad export cookie 777893695268003725 [ 822.591826] LustreError: 30373:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 822.601951] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 822.623665] LustreError: 30373:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 822.872423] Lustre: server umount lustre-MDT0001 complete [ 831.676673] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 834.091955] Key type lgssc unregistered [ 834.393646] LNet: 33519:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 834.404635] LNetError: 33519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 834.418079] LNet: Removed LNI 192.168.203.137@tcp [ 835.333984] Key type .llcrypt unregistered [ 835.335958] Key type ._llcrypt unregistered [ 855.769805] Key type ._llcrypt registered [ 855.773740] Key type .llcrypt registered [ 855.867854] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 856.735668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 856.817344] alg: No test for adler32 (adler32-zlib) [ 857.877985] Lustre: Lustre: Build Version: 2.17.57_84_g5b8c505 [ 858.118899] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 859.783188] Key type lgssc registered [ 861.091058] Lustre: Echo OBD driver; http://www.lustre.org/ [ 868.545356] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 868.595099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 870.178056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 874.647695] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 879.588781] LustreError: 34906:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 879.614737] LustreError: 34906:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 882.447729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 882.879416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 887.037902] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 895.134249] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 900.617199] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 906.898823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 907.306192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 913.715549] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 914.508796] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 920.803630] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 928.737167] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 928.746682] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 928.765521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 929.765426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 929.773078] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 929.790098] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 934.885290] LustreError: 36731:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 935.052952] Lustre: server umount lustre-OST0000 complete [ 939.490848] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 939.502632] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 939.525515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 939.530662] Lustre: Skipped 1 previous similar message [ 943.072468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 943.087367] Lustre: Skipped 2 previous similar messages [ 950.241407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 950.263708] Lustre: Skipped 3 previous similar messages [ 953.311128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 953.557715] Lustre: server umount lustre-MDT0000 complete [ 955.362828] LustreError: 34903:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 955.400474] LustreError: 34903:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 956.940585] LustreError: 34888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788485819 with bad export cookie 2258736508496000097 [ 956.962328] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 957.251211] Lustre: server umount lustre-MDT0001 complete [ 966.716662] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 969.478540] Key type lgssc unregistered [ 969.844629] LNet: 38439:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 969.858370] LNetError: 38439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 969.879900] LNet: Removed LNI 192.168.203.137@tcp [ 970.782899] Key type .llcrypt unregistered [ 970.786660] Key type ._llcrypt unregistered [ 991.393171] Key type ._llcrypt registered [ 991.398281] Key type .llcrypt registered [ 991.533143] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 992.431925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 992.524054] alg: No test for adler32 (adler32-zlib) [ 993.661581] Lustre: Lustre: Build Version: 2.17.57_84_g5b8c505 [ 993.952433] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 995.632357] Key type lgssc registered [ 996.736996] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1003.999323] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1004.027412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1005.470285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1010.030428] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1015.782656] LustreError: 39826:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1015.802194] LustreError: 39826:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1019.168450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1019.473343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1024.258617] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1032.423895] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1040.033672] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1048.563895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1049.042749] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1051.131922] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1055.672694] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1063.839971] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1071.554129] Lustre: lustre-OST0000: Not available for connect from 192.168.203.37@tcp (stopping) [ 1071.584995] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1071.608225] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1074.660933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1074.671053] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1074.672691] Lustre: Skipped 1 previous similar message [ 1074.675846] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1076.636616] Lustre: lustre-OST0000: Not available for connect from 192.168.203.37@tcp (stopping) [ 1076.661539] Lustre: Skipped 1 previous similar message [ 1077.057772] Lustre: server umount lustre-OST0000 complete [ 1097.083604] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1097.091399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1097.111932] Lustre: Skipped 1 previous similar message [ 1097.130234] Lustre: Skipped 1 previous similar message [ 1103.410663] Lustre: server umount lustre-MDT0000 complete [ 1105.378724] LustreError: 39826:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1105.422304] LustreError: 39826:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1107.135493] LustreError: 39808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788485969 with bad export cookie 14823706085569444132 [ 1107.142242] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1107.158138] LustreError: 39808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1113.503913] Lustre: server umount lustre-MDT0001 complete [ 1123.030919] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 1125.533117] Key type lgssc unregistered [ 1125.781580] LNet: 43534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1125.797692] LNetError: 43534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1125.814346] LNet: Removed LNI 192.168.203.137@tcp [ 1126.547503] Key type .llcrypt unregistered [ 1126.552401] Key type ._llcrypt unregistered [ 1149.294646] Key type ._llcrypt registered [ 1149.296843] Key type .llcrypt registered [ 1149.409825] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 1150.216997] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1150.304645] alg: No test for adler32 (adler32-zlib) [ 1151.317351] Lustre: Lustre: Build Version: 2.17.57_84_g5b8c505 [ 1151.579397] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 1153.239218] Key type lgssc registered [ 1154.386969] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1162.199895] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1162.224749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1163.687510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1167.918373] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1173.990405] LustreError: 44943:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1174.020733] LustreError: 44943:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1176.716946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1177.223708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1181.567297] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1189.301372] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1196.485131] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1204.039867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1204.426926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1208.568691] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1211.290928] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1218.450253] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1231.280599] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.37@tcp (stopping) [ 1232.354058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1232.357451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1232.370380] Lustre: Skipped 1 previous similar message [ 1234.839034] Lustre: server umount lustre-MDT0000 complete [ 1236.378189] LustreError: 44938:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1237.472873] LustreError: 45671:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1237.495383] LustreError: 45671:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1238.103833] LustreError: 44925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788486100 with bad export cookie 1377141129589223899 [ 1238.107964] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1238.111967] LustreError: 44925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1241.497038] LustreError: 46981:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1241.542906] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.37@tcp (stopping) [ 1241.550906] Lustre: Skipped 2 previous similar messages [ 1242.593230] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1242.613913] Lustre: Skipped 1 previous similar message [ 1243.618241] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1243.630705] Lustre: Skipped 1 previous similar message [ 1244.865587] Lustre: server umount lustre-MDT0001 complete [ 1259.742981] Lustre: server umount lustre-OST0000 complete [ 1277.058251] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 1280.367876] Key type lgssc unregistered [ 1280.789853] LNet: 48803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1280.795321] LNetError: 48803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1280.818833] LNet: Removed LNI 192.168.203.137@tcp [ 1281.788169] Key type .llcrypt unregistered [ 1281.790975] Key type ._llcrypt unregistered [ 1307.395163] Key type ._llcrypt registered [ 1307.397691] Key type .llcrypt registered [ 1307.563109] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 1308.897483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1308.950546] alg: No test for adler32 (adler32-zlib) [ 1310.009700] Lustre: Lustre: Build Version: 2.17.57_84_g5b8c505 [ 1310.223391] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 1311.871195] Key type lgssc registered [ 1313.106557] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1320.683890] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1320.695136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1336.223446] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1341.343303] LustreError: 50189:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.137@tcp: failed processing log, type 4: rc = -110 [ 1368.095695] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1374.382706] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1382.792927] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1549.199836] Lustre: server umount lustre-OST0000 complete [ 1568.240713] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 1571.063200] Key type lgssc unregistered [ 1571.373783] LNet: 51866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1571.380400] LNetError: 51866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1571.404135] LNet: Removed LNI 192.168.203.137@tcp [ 1572.311188] Key type .llcrypt unregistered [ 1572.314620] Key type ._llcrypt unregistered [ 1595.215249] Key type ._llcrypt registered [ 1595.217440] Key type .llcrypt registered [ 1595.335290] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 1596.383895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1596.412985] alg: No test for adler32 (adler32-zlib) [ 1597.403671] Lustre: Lustre: Build Version: 2.17.57_84_g5b8c505 [ 1597.649149] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 1599.303352] Key type lgssc registered [ 1600.455634] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1607.845900] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1607.868538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1609.292606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1614.094689] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1618.412645] LustreError: 53273:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1618.438221] LustreError: 53273:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1622.824121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1623.236881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1627.767486] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1634.926921] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1640.679036] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1647.281191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1647.756732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1653.754881] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1654.026337] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1661.515880] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1664.480512] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1664.483933] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1664.495436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1664.514952] Lustre: Skipped 1 previous similar message [ 1664.536956] Lustre: Skipped 1 previous similar message [ 1669.614721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1669.619634] Lustre: Skipped 1 previous similar message [ 1670.474883] Lustre: server umount lustre-OST0000 complete [ 1679.843177] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1679.851599] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1679.856380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1679.871549] Lustre: Skipped 1 previous similar message [ 1681.285287] Lustre: server umount lustre-MDT0000 complete [ 1684.966230] LustreError: 54002:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1684.976047] LustreError: 54002:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1685.646022] LustreError: 53254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788486548 with bad export cookie 7945161192967381286 [ 1685.649161] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1685.666429] LustreError: 53254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1686.158877] Lustre: server umount lustre-MDT0001 complete [ 1698.057738] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 1701.499345] Key type lgssc unregistered [ 1701.888241] LNet: 56804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1701.902535] LNetError: 56804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1701.923655] LNet: Removed LNI 192.168.203.137@tcp [ 1702.881571] Key type .llcrypt unregistered [ 1702.887985] Key type ._llcrypt unregistered [ 1726.953792] Key type ._llcrypt registered [ 1726.957664] Key type .llcrypt registered [ 1727.113669] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 1728.553277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1728.569988] alg: No test for adler32 (adler32-zlib) [ 1729.611920] Lustre: Lustre: Build Version: 2.17.57_84_g5b8c505 [ 1729.868143] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 1731.583166] Key type lgssc registered [ 1732.630644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1740.756807] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1740.775451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1756.320393] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1761.439486] LustreError: 58192:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.137@tcp: failed processing log, type 4: rc = -110 [ 1788.127184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1792.999650] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1799.450987] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1805.238798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1805.721943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1805.792583] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1809.861470] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1810.917626] LustreError: 58217:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1810.951541] LustreError: 58217:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1816.045239] LustreError: 58218:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1818.865749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1819.283300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1824.100451] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1831.507808] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1836.385766] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1839.587466] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1839.598260] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1839.630223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1841.634926] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1841.649132] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1841.661319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1843.174047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1843.184813] Lustre: Skipped 1 previous similar message [ 1844.779818] Lustre: server umount lustre-OST0000 complete [ 1865.760495] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1865.778039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1868.833096] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1872.103113] Lustre: server umount lustre-MDT0000 complete [ 1873.894623] LustreError: 60578:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1873.916127] LustreError: 60578:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1876.014546] LustreError: 59968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788486738 with bad export cookie 2643660035557934143 [ 1876.017171] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1876.032041] LustreError: 59968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1882.443130] Lustre: server umount lustre-MDT0001 complete [ 1894.161562] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 1897.060975] Key type lgssc unregistered [ 1897.412932] LNet: 61930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1897.418051] LNetError: 61930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1897.434973] LNet: Removed LNI 192.168.203.137@tcp [ 1898.487190] Key type .llcrypt unregistered [ 1898.489351] Key type ._llcrypt unregistered [ 1922.359451] Key type ._llcrypt registered [ 1922.361689] Key type .llcrypt registered [ 1922.442165] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 1923.221527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1923.271673] alg: No test for adler32 (adler32-zlib) [ 1924.370504] Lustre: Lustre: Build Version: 2.17.57_84_g5b8c505 [ 1924.584322] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 1926.248478] Key type lgssc registered [ 1927.292200] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1934.097240] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1934.129142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1935.616953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1940.040781] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1946.084146] LustreError: 63338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1946.107550] LustreError: 63338:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1947.935437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1948.352274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1952.002934] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1958.330461] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1963.147514] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1969.311313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1969.706263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1975.482524] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1980.996954] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1982.521394] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1999.333134] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1999.350936] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1999.360136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2000.605988] Lustre: server umount lustre-OST0000 complete [ 2004.960865] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2004.980699] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2005.017509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2006.497695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2006.499645] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2011.627541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2011.639496] Lustre: Skipped 2 previous similar messages [ 2013.730585] Lustre: server umount lustre-MDT0000 complete [ 2016.740135] LustreError: 63334:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2016.768846] LustreError: 63334:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2016.800723] LustreError: 63320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788486879 with bad export cookie 14194278717814397807 [ 2016.802413] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2016.816743] LustreError: 63320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2017.139633] Lustre: server umount lustre-MDT0001 complete [ 2027.573165] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 2031.348352] Key type lgssc unregistered [ 2031.674031] LNet: 66889:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2031.683965] LNetError: 66889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2031.702157] LNet: Removed LNI 192.168.203.137@tcp [ 2032.904123] Key type .llcrypt unregistered [ 2032.905943] Key type ._llcrypt unregistered [ 2057.396387] Key type ._llcrypt registered [ 2057.398697] Key type .llcrypt registered [ 2057.474249] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 2058.459567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2058.518376] alg: No test for adler32 (adler32-zlib) [ 2059.577657] Lustre: Lustre: Build Version: 2.17.57_84_g5b8c505 [ 2059.762480] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 2061.439196] Key type lgssc registered [ 2062.684668] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2071.083448] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 21:55:31 (1788486931) [ 2077.533741] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2077.549144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2079.044568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2083.779986] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2088.421130] LustreError: 68592:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2088.438202] LustreError: 68592:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2093.015545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2093.362687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2097.871852] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2104.874597] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2109.934985] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2115.956568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2116.319724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2122.026714] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2123.509696] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2128.402846] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2133.133296] Lustre: Failing over lustre-MDT0000 [ 2133.403834] Lustre: server umount lustre-MDT0000 complete [ 2133.985862] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2134.002240] LustreError: 69320:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2134.009524] Lustre: Skipped 1 previous similar message [ 2134.032635] LustreError: 69320:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2139.108798] LustreError: 68592:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2139.123565] LustreError: 68592:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2142.082975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2142.211610] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2142.360338] Lustre: *** cfs_fail_loc=135, val=0*** [ 2142.470345] LustreError: 71675:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2142.480929] LustreError: 71675:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.137@tcp: cfg command failed: rc = -2 [ 2142.495880] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2142.495880] [ 2142.503356] LustreError: MGC192.168.203.137@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2142.514932] LustreError: 71664:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2142.529940] LustreError: 71664:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 2142.545063] LustreError: 71664:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2142.555188] LustreError: 71664:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2142.571735] Lustre: server umount lustre-MDT0000 complete [ 2142.574523] LustreError: 71664:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2144.226487] LustreError: 68591:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2144.235153] LustreError: 68591:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2148.837371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2148.968925] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2149.272730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2149.332725] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2150.297232] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2153.185921] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2154.484381] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2154.509494] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2154.536552] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2160.097073] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2160.107871] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2160.120585] Lustre: Skipped 1 previous similar message [ 2160.130839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2162.048799] Lustre: server umount lustre-OST0000 complete [ 2169.825181] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2169.837111] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2169.841401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2169.842791] Lustre: Skipped 1 previous similar message [ 2172.062504] Lustre: server umount lustre-MDT0000 complete [ 2174.948676] LustreError: 68586:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2174.968645] LustreError: 68586:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2175.477567] LustreError: 68572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788487038 with bad export cookie 10651178093910653885 [ 2175.480929] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2175.501142] LustreError: 68572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2185.906483] Lustre: server umount lustre-MDT0001 complete [ 2195.910908] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 2198.353948] Key type lgssc unregistered [ 2198.703952] LNet: 73451:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2198.719678] LNetError: 73451:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2198.735479] LNet: Removed LNI 192.168.203.137@tcp [ 2199.651136] Key type .llcrypt unregistered [ 2199.655572] Key type ._llcrypt unregistered [ 2219.129688] Key type ._llcrypt registered [ 2219.132372] Key type .llcrypt registered [ 2219.258638] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 2220.218238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2220.251042] alg: No test for adler32 (adler32-zlib) [ 2221.309301] Lustre: Lustre: Build Version: 2.17.57_84_g5b8c505 [ 2221.494849] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 2223.167909] Key type lgssc registered [ 2224.393747] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2232.531333] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2232.573308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2233.999157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2238.488913] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2243.558274] LustreError: 74841:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2243.584597] LustreError: 74841:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2246.202266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2246.562315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2250.590790] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2257.390971] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2262.516070] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2269.354524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2269.774184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2276.367776] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2279.030363] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2283.918616] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2290.107874] Lustre: Failing over lustre-MDT0000 [ 2290.428786] Lustre: server umount lustre-MDT0000 complete [ 2291.134175] LustreError: 76180:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2292.704338] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2292.711782] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2292.726385] LustreError: 75568:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2294.761238] LustreError: 74841:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2294.770415] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2294.818653] Lustre: Skipped 1 previous similar message [ 2299.874418] LustreError: 74836:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2299.893260] LustreError: 74836:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2301.379322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2301.530924] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2301.675299] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2301.836516] LustreError: 77925:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2301.841332] LustreError: 77925:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.137@tcp: cfg command failed: rc = -9 [ 2301.848196] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2301.848196] [ 2301.856240] LustreError: MGC192.168.203.137@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2301.865846] LustreError: 77914:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2301.870724] LustreError: 77914:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -9 [ 2301.878114] LustreError: 77914:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2301.882734] LustreError: 77914:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2301.914563] Lustre: server umount lustre-MDT0000 complete [ 2301.917026] LustreError: 77914:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2310.120910] LustreError: 74841:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2310.172332] LustreError: 74841:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 2312.057388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2312.447650] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2312.470749] LustreError: 74832:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2312.965566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2313.045548] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2316.700550] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2318.325704] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2318.365961] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2318.518093] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2319.383363] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2333.667150] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2333.667705] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2333.676940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2333.951972] Lustre: Skipped 1 previous similar message [ 2338.792152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2338.805081] Lustre: Skipped 1 previous similar message [ 2339.077283] Lustre: server umount lustre-OST0000 complete [ 2343.906662] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2343.913770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2343.929693] Lustre: Skipped 1 previous similar message [ 2343.954865] Lustre: Skipped 2 previous similar messages [ 2349.024493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2349.046669] Lustre: Skipped 1 previous similar message [ 2350.013923] Lustre: server umount lustre-MDT0000 complete [ 2353.220488] LustreError: 74820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788487215 with bad export cookie 12764556173709668621 [ 2353.231459] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2353.239613] LustreError: 74820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2353.520182] Lustre: server umount lustre-MDT0001 complete [ 2365.041491] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 2367.822872] Key type lgssc unregistered [ 2368.068945] LNet: 79707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2368.078383] LNetError: 79707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2368.095727] LNet: Removed LNI 192.168.203.137@tcp [ 2368.820929] Key type .llcrypt unregistered [ 2368.824520] Key type ._llcrypt unregistered [ 2380.510437] Key type ._llcrypt registered [ 2380.512598] Key type .llcrypt registered [ 2380.595583] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 2380.819303] Key type .llcrypt unregistered [ 2380.821157] Key type ._llcrypt unregistered [ 2402.470687] Key type ._llcrypt registered [ 2402.473974] Key type .llcrypt registered [ 2402.557846] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 2403.390491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2403.411632] alg: No test for adler32 (adler32-zlib) [ 2404.428905] Lustre: Lustre: Build Version: 2.17.57_84_g5b8c505 [ 2404.591464] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 2406.255141] Key type lgssc registered [ 2407.171082] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2413.956246] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2413.983028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2415.354470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2419.486911] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2425.825961] LustreError: 81623:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2425.840261] LustreError: 81623:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2427.310283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2427.651230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2431.552976] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2438.036449] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2443.261340] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2450.878603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2451.202521] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2458.285486] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2460.417555] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2465.544301] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2473.955370] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2473.967449] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2473.987327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2476.002234] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2476.009430] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2476.011510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2476.040142] Lustre: Skipped 1 previous similar message [ 2479.930505] Lustre: server umount lustre-OST0000 complete [ 2484.194493] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2484.202610] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2484.218944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2486.240409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2486.242196] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2491.361373] LustreError: 81623:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2491.397133] LustreError: 81623:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2491.914897] Lustre: server umount lustre-MDT0000 complete [ 2495.373885] LustreError: 81603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788487358 with bad export cookie 3288297727208697654 [ 2495.379726] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2495.386984] LustreError: 81603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2495.731484] Lustre: server umount lustre-MDT0001 complete [ 2506.072443] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 2508.866719] Key type lgssc unregistered [ 2509.183953] LNet: 85154:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2509.194407] LNetError: 85154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2509.209507] LNet: Removed LNI 192.168.203.137@tcp [ 2509.953957] Key type .llcrypt unregistered [ 2509.960660] Key type ._llcrypt unregistered [ 2530.845322] Key type ._llcrypt registered [ 2530.847449] Key type .llcrypt registered [ 2530.935408] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 2531.842926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2531.868420] alg: No test for adler32 (adler32-zlib) [ 2532.882942] Lustre: Lustre: Build Version: 2.17.57_84_g5b8c505 [ 2533.062666] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 2534.759204] Key type lgssc registered [ 2535.706189] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2542.509262] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2542.538033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2544.068125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2548.416619] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2553.318268] LustreError: 86556:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2553.355030] LustreError: 86556:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2556.194941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2556.449624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2560.594042] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2567.956822] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2573.625376] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2580.742385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2581.166855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2587.734832] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2588.357511] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2595.340600] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2602.984373] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2602.997084] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2603.015543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2604.011130] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2604.012501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2604.025199] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2604.030179] Lustre: Skipped 1 previous similar message [ 2607.153775] Lustre: server umount lustre-OST0000 complete [ 2612.706859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2612.714447] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2612.754914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2617.443496] Lustre: server umount lustre-MDT0000 complete [ 2619.364103] LustreError: 89045:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2619.387743] LustreError: 89045:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2621.452059] LustreError: 86536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788487484 with bad export cookie 16838770221370096365 [ 2621.458804] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2621.465060] LustreError: 86536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2622.006025] Lustre: server umount lustre-MDT0001 complete [ 2633.444377] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 2636.495870] Key type lgssc unregistered [ 2636.844673] LNet: 90087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2636.852693] LNetError: 90087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2636.875618] LNet: Removed LNI 192.168.203.137@tcp [ 2637.927322] Key type .llcrypt unregistered [ 2637.937030] Key type ._llcrypt unregistered [ 2661.332763] Key type ._llcrypt registered [ 2661.335374] Key type .llcrypt registered [ 2661.438098] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 2662.415386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2662.443109] alg: No test for adler32 (adler32-zlib) [ 2663.779226] Lustre: Lustre: Build Version: 2.17.57_84_g5b8c505 [ 2664.051110] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 2665.783579] Key type lgssc registered [ 2667.086226] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2674.713208] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2674.733389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2676.207466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2680.738834] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2685.414908] LustreError: 91497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2685.436387] LustreError: 91497:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2690.403495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2690.528535] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2690.725879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2695.570546] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2703.387553] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2709.462497] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2716.159589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2716.624146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2721.790639] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2723.743542] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2731.268221] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2743.269790] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2743.289893] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2743.309572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2745.322815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2745.323442] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2748.617066] Lustre: server umount lustre-OST0000 complete [ 2752.482870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2752.502123] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2752.527758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2752.534554] Lustre: Skipped 1 previous similar message [ 2755.563695] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2758.585572] Lustre: server umount lustre-MDT0000 complete [ 2760.672695] LustreError: 91492:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2760.692633] LustreError: 91492:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2762.337101] LustreError: 91478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788487625 with bad export cookie 13353759123082121050 [ 2762.357779] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2762.365070] LustreError: 91478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2762.753682] Lustre: server umount lustre-MDT0001 complete [ 2773.705901] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 2777.250806] Key type lgssc unregistered [ 2777.570036] LNet: 95031:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2777.581764] LNetError: 95031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2777.602928] LNet: Removed LNI 192.168.203.137@tcp [ 2778.486736] Key type .llcrypt unregistered [ 2778.492516] Key type ._llcrypt unregistered [ 2803.866365] Key type ._llcrypt registered [ 2803.868492] Key type .llcrypt registered [ 2804.010795] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 2805.146160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2805.166965] alg: No test for adler32 (adler32-zlib) [ 2806.233820] Lustre: Lustre: Build Version: 2.17.57_84_g5b8c505 [ 2806.454309] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 2808.175947] Key type lgssc registered [ 2809.206739] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2816.329332] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2816.375882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2832.095418] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2837.216457] LustreError: 96413:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.137@tcp: failed processing log, type 4: rc = -110 [ 2863.903569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2870.266385] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2877.710619] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2883.415748] Lustre: server umount lustre-OST0000 complete [ 2891.335465] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 22:09:11 (1788487751) [ 2897.692223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2898.168119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2902.753703] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2903.526445] LustreError: 98050:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2903.546604] LustreError: 98050:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2907.619282] LustreError: 98051:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2911.103685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2911.384234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2915.517345] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2922.447905] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2927.335363] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2933.728755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2934.119434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2940.633513] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2942.394021] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 2948.360934] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2962.916917] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2962.917749] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2962.929794] LustreError: Skipped 1 previous similar message [ 2962.930507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2962.955377] Lustre: Skipped 1 previous similar message [ 2965.572859] Lustre: server umount lustre-OST0000 complete [ 2973.153133] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2973.155294] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2973.160575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2973.183850] Lustre: Skipped 1 previous similar message [ 2973.201200] Lustre: Skipped 2 previous similar messages [ 2975.005825] Lustre: server umount lustre-MDT0000 complete [ 2978.272548] LustreError: 98051:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2978.312827] LustreError: 98051:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2978.388635] LustreError: 98786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788487841 with bad export cookie 17067512993696203205 [ 2978.389894] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2978.399955] LustreError: 98786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2978.759167] Lustre: server umount lustre-MDT0001 complete [ 2989.725207] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 2992.151794] Key type lgssc unregistered [ 2992.418603] LNet: 101739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2992.424377] LNetError: 101739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2992.436480] LNet: Removed LNI 192.168.203.137@tcp [ 2993.141470] Key type .llcrypt unregistered [ 2993.146695] Key type ._llcrypt unregistered [ 3033.449371] Key type ._llcrypt registered [ 3033.451784] Key type .llcrypt registered [ 3033.556762] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 3034.628431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3034.728350] alg: No test for adler32 (adler32-zlib) [ 3035.756637] Lustre: Lustre: Build Version: 2.17.57_84_g5b8c505 [ 3035.962399] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 3037.615244] Key type lgssc registered [ 3038.712518] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3045.515596] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3045.535205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3046.894877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3050.467151] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3057.122748] LustreError: 103937:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3057.134746] LustreError: 103937:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3059.952923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3060.218451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3065.596222] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3073.329347] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3079.699412] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3086.670945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3087.047422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3092.224142] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3092.931580] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3100.147787] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3112.929928] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3112.931788] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3112.937532] LustreError: Skipped 1 previous similar message [ 3112.952184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3114.983236] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3114.984510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3115.004069] Lustre: Skipped 1 previous similar message [ 3116.207473] Lustre: server umount lustre-OST0000 complete [ 3120.101179] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3120.114957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3120.120821] Lustre: Skipped 1 previous similar message [ 3125.220805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3125.231690] Lustre: Skipped 1 previous similar message [ 3125.512799] Lustre: server umount lustre-MDT0000 complete [ 3128.905557] LustreError: 103917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788487991 with bad export cookie 13079993006971248281 [ 3128.913159] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3128.916942] LustreError: 103917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3129.171391] Lustre: server umount lustre-MDT0001 complete [ 3139.115448] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 3141.593809] Key type lgssc unregistered [ 3141.860076] LNet: 107467:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3141.878297] LNetError: 107467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3141.915189] LNet: Removed LNI 192.168.203.137@tcp [ 3142.709075] Key type .llcrypt unregistered [ 3142.711265] Key type ._llcrypt unregistered [ 3160.389753] Key type ._llcrypt registered [ 3160.392021] Key type .llcrypt registered [ 3160.498160] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 3161.888149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3161.944401] alg: No test for adler32 (adler32-zlib) [ 3163.116159] Lustre: Lustre: Build Version: 2.17.57_84_g5b8c505 [ 3163.367669] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 3165.051211] Key type lgssc registered [ 3166.268960] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3174.208909] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3174.226840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3189.727565] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3194.850680] LustreError: 108707:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.137@tcp: failed processing log, type 4: rc = -110 [ 3221.471318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3227.243724] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3234.668721] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3241.181657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3241.487953] LustreError: MGC192.168.203.137@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3241.496271] LustreError: 109730:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3241.500882] LustreError: 109730:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 3241.508160] LustreError: 109730:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 3241.513072] LustreError: 108715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788488104 with bad export cookie 12081886037325543907 [ 3241.523286] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3243.636403] Lustre: server umount lustre-MDT0000 complete [ 3243.639725] LustreError: 109730:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3269.725715] Lustre: server umount lustre-OST0000 complete [ 3280.130654] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 3291.171487] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 3329.859637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3329.993579] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3330.014071] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3330.102184] Lustre: lustre-MDT0000: new disk, initializing [ 3330.151467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3330.162783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3334.983038] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3345.809474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3345.997266] Lustre: 112524:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3346.069895] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3346.073345] Lustre: Skipped 1 previous similar message [ 3346.132103] Lustre: lustre-MDT0001: new disk, initializing [ 3346.232073] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3346.282158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3346.298674] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3350.919277] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3359.823492] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3365.265703] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3372.262359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3372.515902] Lustre: lustre-OST0000: new disk, initializing [ 3372.521225] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3372.531639] Lustre: 114493:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3372.634262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3378.267547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3378.282294] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3378.416830] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3378.759816] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3388.398111] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3393.278463] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3393.486330] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3397.597330] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3397.870618] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3404.260178] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3404.269855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3404.283225] Lustre: Skipped 1 previous similar message [ 3404.305506] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3405.905918] Lustre: server umount lustre-OST0000 complete [ 3409.378053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3409.394272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3409.405214] Lustre: Skipped 2 previous similar messages [ 3414.500293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3414.504798] Lustre: Skipped 1 previous similar message [ 3419.616271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3419.625517] Lustre: Skipped 1 previous similar message [ 3423.199143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3423.361475] Lustre: server umount lustre-MDT0000 complete [ 3426.849654] LustreError: 113329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788488289 with bad export cookie 12081886037325544012 [ 3426.857485] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3426.863757] LustreError: 113329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3427.218958] Lustre: server umount lustre-MDT0001 complete [ 3434.858405] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 22:18:15 (1788488295) [ 3437.635498] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3452.173151] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 3460.998555] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 3493.201823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3493.424184] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3493.463470] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3493.569734] Lustre: lustre-MDT0000: new disk, initializing [ 3493.678749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3493.705688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3498.700434] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3509.224796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3509.394889] Lustre: 119392:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3509.498276] LustreError: 119404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3509.505930] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3509.524769] LustreError: 119404:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3509.529718] Lustre: Skipped 1 previous similar message [ 3509.615118] Lustre: lustre-MDT0001: new disk, initializing [ 3509.737502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3509.773205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3509.796190] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3514.579596] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3523.941427] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3529.593143] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3535.274153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3535.599560] Lustre: lustre-OST0000: new disk, initializing [ 3535.603902] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3535.613444] Lustre: 121358:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3535.673982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3541.086454] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3544.658609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3544.681476] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3544.834395] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3549.963741] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3554.626203] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3554.883555] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3558.982527] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3559.201522] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3565.547248] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3565.552304] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3565.556446] LustreError: Skipped 1 previous similar message [ 3565.573614] Lustre: Skipped 1 previous similar message [ 3565.574646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3565.590721] Lustre: Skipped 1 previous similar message [ 3575.263807] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3575.431437] Lustre: server umount lustre-OST0000 complete [ 3580.900861] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3580.905524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3580.916614] Lustre: Skipped 2 previous similar messages [ 3580.922287] Lustre: Skipped 2 previous similar messages [ 3584.784393] Lustre: server umount lustre-MDT0000 complete [ 3586.030670] LustreError: 119401:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3586.057042] LustreError: 119401:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3589.045661] LustreError: 119384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788488451 with bad export cookie 12081886037325544894 [ 3589.062544] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3589.077139] LustreError: 119384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3589.722709] Lustre: server umount lustre-MDT0001 complete [ 3601.192024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3601.818232] LustreError: 123574:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3602.008087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3607.009768] LustreError: 123575:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3608.131962] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3612.129869] LustreError: 123574:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3616.221944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3620.703923] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3628.190362] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3633.638851] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3639.855478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3646.794531] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3654.158409] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3661.494611] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3666.913802] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3666.929289] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3666.959924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3666.974585] Lustre: Skipped 1 previous similar message [ 3666.982419] Lustre: Skipped 1 previous similar message [ 3671.763055] Lustre: server umount lustre-OST0000 complete [ 3677.153665] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3677.184558] Lustre: Skipped 1 previous similar message [ 3681.929000] Lustre: server umount lustre-MDT0000 complete [ 3682.272366] LustreError: 123575:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3682.283654] LustreError: 123575:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3685.543464] LustreError: 123557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788488548 with bad export cookie 12081886037325545776 [ 3685.557683] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3685.566035] LustreError: 123557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3685.913488] Lustre: server umount lustre-MDT0001 complete [ 3695.978511] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 3698.753776] Key type lgssc unregistered [ 3699.073807] LNet: 127205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3699.088332] LNetError: 127205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3699.110071] LNet: Removed LNI 192.168.203.137@tcp [ 3700.047954] Key type .llcrypt unregistered [ 3700.050370] Key type ._llcrypt unregistered [ 3717.827876] Key type ._llcrypt registered [ 3717.829926] Key type .llcrypt registered [ 3717.964507] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 3734.123300] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 3735.480337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3735.503054] alg: No test for adler32 (adler32-zlib) [ 3736.708472] Lustre: Lustre: Build Version: 2.17.57_84_g5b8c505 [ 3736.973224] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 3738.735200] Key type lgssc registered [ 3739.919592] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3772.444120] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3772.467200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3773.809960] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3773.843224] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3773.976496] Lustre: lustre-MDT0000: new disk, initializing [ 3774.049039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3774.109234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3779.631504] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3792.339360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3792.486786] Lustre: 130354:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3792.617132] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3792.628693] Lustre: Skipped 1 previous similar message [ 3792.743558] Lustre: lustre-MDT0001: new disk, initializing [ 3792.874417] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3792.945416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3792.967648] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3797.561360] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3806.755991] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3813.079971] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3821.336718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3821.757945] Lustre: lustre-OST0000: new disk, initializing [ 3821.761986] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3821.769065] Lustre: 132324:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3821.852837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3829.574193] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3830.814520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3830.821256] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3830.889774] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3839.160053] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3844.346680] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3844.525541] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3848.093287] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3848.351045] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3851.244337] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3851.245529] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3851.248916] LustreError: Skipped 1 previous similar message [ 3851.260956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3851.270583] Lustre: Skipped 1 previous similar message [ 3856.355485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3856.668422] Lustre: server umount lustre-OST0000 complete [ 3861.474163] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3861.493159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3861.503199] Lustre: Skipped 2 previous similar messages [ 3861.539595] Lustre: Skipped 2 previous similar messages [ 3866.595384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3866.606558] Lustre: Skipped 1 previous similar message [ 3871.713981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3871.730175] Lustre: Skipped 1 previous similar message [ 3875.295296] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3875.648759] Lustre: server umount lustre-MDT0000 complete [ 3880.877041] LustreError: 130347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788488743 with bad export cookie 3610335969300509945 [ 3880.885644] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3880.886811] LustreError: 130347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3881.205331] Lustre: server umount lustre-MDT0001 complete [ 3887.911883] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 22:25:48 (1788488748) [ 3893.590686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3894.121110] LustreError: 134736:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3894.145079] LustreError: 134736:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3894.195646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3898.076566] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3899.364686] LustreError: 134737:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3903.462761] LustreError: 134736:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3904.672175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3905.126302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3910.560435] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3918.061533] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3924.019165] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3926.175257] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3926.191500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3932.547826] Lustre: server umount lustre-MDT0000 complete [ 3935.726758] LustreError: 134733:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3935.764679] LustreError: 134733:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3936.129727] LustreError: 134717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788488798 with bad export cookie 3610335969300510827 [ 3936.137079] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3936.145239] LustreError: 134717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3936.662737] Lustre: server umount lustre-MDT0001 complete [ 3945.138268] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 22:26:45 (1788488805) [ 3952.033029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3967.647646] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3972.768059] LustreError: 137151:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.137@tcp: failed processing log, type 4: rc = -110 [ 3998.560590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4005.327938] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4012.374350] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4014.642601] Lustre: server umount lustre-OST0000 complete [ 4023.427574] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 22:28:03 (1788488883) [ 4030.181723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4030.582575] LustreError: 138597:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4030.658078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4035.280240] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4041.188386] LustreError: 138598:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4041.208019] LustreError: 138598:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4043.953736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4048.943053] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4056.802380] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4062.427281] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4069.686172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4070.148150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4070.162853] Lustre: Skipped 1 previous similar message [ 4076.409463] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4084.893699] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4099.049606] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4099.073099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4099.090901] Lustre: Skipped 3 previous similar messages [ 4099.103926] Lustre: Skipped 5 previous similar messages [ 4112.351265] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4112.571254] Lustre: server umount lustre-MDT0000 complete [ 4114.406976] LustreError: 140767:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4114.425077] LustreError: 140767:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4116.578318] LustreError: 138579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788488979 with bad export cookie 3610335969300511492 [ 4116.600452] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4117.111951] Lustre: server umount lustre-MDT0001 complete [ 4131.828693] Lustre: server umount lustre-OST0000 complete [ 4140.568466] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 22:30:00 (1788489000) [ 4147.870965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4148.530857] LustreError: 142272:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4148.674129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4153.365514] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4162.275150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4167.710347] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4175.183383] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4181.931525] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4189.491241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4193.904559] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4196.087596] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4203.168766] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4207.927304] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4208.144382] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4212.828821] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4213.005038] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4218.849805] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4218.851531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4218.887289] Lustre: Skipped 1 previous similar message [ 4218.904404] Lustre: Skipped 6 previous similar messages [ 4218.907866] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4221.262607] Lustre: server umount lustre-OST0000 complete [ 4229.091063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4229.094122] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4231.560091] Lustre: server umount lustre-MDT0000 complete [ 4234.215412] LustreError: 143008:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4234.244123] LustreError: 143008:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4234.600458] LustreError: 142252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788489097 with bad export cookie 3610335969300512563 [ 4234.602053] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4234.618094] LustreError: 142252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4234.895951] Lustre: server umount lustre-MDT0001 complete [ 4242.420697] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 22:31:42 (1788489102) [ 4248.921087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4264.479954] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4269.601607] LustreError: 146443:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.137@tcp: failed processing log, type 4: rc = -110 [ 4295.393810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4295.403525] Lustre: Skipped 2 previous similar messages [ 4301.077828] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4308.717679] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4316.325022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4317.051805] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4321.918599] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4330.787154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4335.711610] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4343.884294] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4349.521680] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4354.436979] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4354.710643] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4360.035334] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4360.387689] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4366.819659] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4366.831283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4366.833790] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4366.841111] Lustre: Skipped 3 previous similar messages [ 4366.866330] Lustre: Skipped 5 previous similar messages [ 4368.892862] Lustre: server umount lustre-MDT0000 complete [ 4371.941596] LustreError: 147481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4371.955376] LustreError: 147481:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4372.896897] LustreError: 146450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788489235 with bad export cookie 3610335969300513585 [ 4372.906664] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4372.910707] LustreError: 146450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4373.292886] Lustre: server umount lustre-MDT0001 complete [ 4387.402768] Lustre: server umount lustre-OST0000 complete [ 4395.383897] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 22:34:15 (1788489255) [ 4401.678412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4417.379362] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4422.495460] LustreError: 150640:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.137@tcp: failed processing log, type 4: rc = -110 [ 4453.437625] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4460.030799] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4465.845664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4466.348244] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4469.829238] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4478.048192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4481.640288] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4490.035844] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4498.334733] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4506.661552] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4506.932841] Lustre: lustre-OST0001: new disk, initializing [ 4506.940309] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4506.957653] Lustre: 153512:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4512.418838] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4515.350690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4515.362370] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4515.471586] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4520.584746] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4524.849424] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4525.037713] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4529.191077] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4529.396412] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4531.177486] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4531.180614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4531.203396] Lustre: Skipped 1 previous similar message [ 4531.213870] Lustre: Skipped 1 previous similar message [ 4537.088044] Lustre: server umount lustre-OST0000 complete [ 4540.898248] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4546.414752] Lustre: server umount lustre-OST0001 complete [ 4550.114724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4550.121576] LustreError: Skipped 1 previous similar message [ 4555.842552] Lustre: server umount lustre-MDT0000 complete [ 4559.023291] LustreError: 150646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788489421 with bad export cookie 3610335969300514537 [ 4559.028827] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4559.035140] LustreError: 150646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4559.435611] Lustre: server umount lustre-MDT0001 complete [ 4579.329418] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 22:37:19 (1788489439) [ 4580.495383] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4581.778559] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4583.108414] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 22:37:23 (1788489443) [ 4588.547852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4588.816853] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4588.852854] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4589.066655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4589.076259] Lustre: Skipped 6 previous similar messages [ 4594.012089] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4602.114596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4602.275615] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4602.348398] Lustre: 157071:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4607.044898] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4614.390463] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4619.267318] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4625.464635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4625.607265] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4631.696654] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4632.219924] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4639.321261] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4643.626974] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4643.818357] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4647.733650] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4647.916559] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4652.000262] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4652.004373] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4652.015751] Lustre: Skipped 4 previous similar messages [ 4655.821771] Lustre: server umount lustre-OST0000 complete [ 4676.385275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4677.735611] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4677.749395] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4681.727196] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4688.282218] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4693.099904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4693.124792] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4694.174489] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4694.313760] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4698.873487] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4699.038371] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4704.357805] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9e7cc586a000.ost_server_uuid 50 [ 4706.044874] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9e7cc586a000.ost_server_uuid in FULL state after 0 sec [ 4706.306852] LustreError: 157076:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.37@tcp) returned error from blocking AST (req@00000000c9d85afb x1875366221448192 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000073d6802d/0x321a7d6f89f91043 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.37@tcp remote: 0xfdd8f77530a8048d expref: 5 pid: 157076 timeout: 4806 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4706.385127] LustreError: lustre-MDT0000: A client on nid 192.168.203.37@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4706.400359] LustreError: 157066:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.37@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000073d6802d/0x321a7d6f89f91043 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.37@tcp remote: 0xfdd8f77530a8048d expref: 6 pid: 157076 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4711.915512] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4711.918464] LustreError: Skipped 1 previous similar message [ 4717.683879] Lustre: server umount lustre-OST0000 complete [ 4727.841539] Lustre: server umount lustre-MDT0000 complete [ 4729.315513] LustreError: 158940:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4729.349704] LustreError: 158940:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 4731.418609] LustreError: 157062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788489594 with bad export cookie 3610335969300515720 [ 4731.422344] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4731.437392] LustreError: 157062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4737.783771] Lustre: server umount lustre-MDT0001 complete [ 4748.068341] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 4751.427769] Key type lgssc unregistered [ 4751.739754] LNet: 163184:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4751.748880] LNetError: 163184:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4751.759907] LNet: Removed LNI 192.168.203.137@tcp [ 4752.594993] Key type .llcrypt unregistered [ 4752.599818] Key type ._llcrypt unregistered [ 4775.704297] Key type ._llcrypt registered [ 4775.707070] Key type .llcrypt registered [ 4775.798147] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 4776.815184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4776.939466] alg: No test for adler32 (adler32-zlib) [ 4777.973672] Lustre: Lustre: Build Version: 2.17.57_84_g5b8c505 [ 4778.210805] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 4779.890259] Key type lgssc registered [ 4780.927889] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4787.918555] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4787.952973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4789.540362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4793.764248] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4799.970621] LustreError: 164592:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4799.987369] LustreError: 164592:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4801.069831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4801.401519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4805.009875] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4811.246501] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4817.398994] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4823.660397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4823.943743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4829.641471] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4835.251740] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4836.444347] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4843.202613] Lustre: Failing over lustre-MDT0000 [ 4843.646915] Lustre: server umount lustre-MDT0000 complete [ 4844.466190] LustreError: 164587:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4845.552317] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4845.565184] LustreError: 165321:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4845.580033] Lustre: Skipped 1 previous similar message [ 4850.658247] LustreError: 164589:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4850.685966] LustreError: 164589:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4855.780336] LustreError: 165679:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4855.802744] LustreError: 165679:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4855.915513] LustreError: 164573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.37@tcp arrived at 1788489718 with bad export cookie 16426468222894798183 [ 4855.936668] LustreError: 164573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4861.893296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4861.919152] Lustre: 163902:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788489708/real 1788489708] req@0000000031ab0094 x1875367312777472/t0(0) o400->MGC192.168.203.137@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788489724 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4861.941489] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4867.050141] LustreError: 164592:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4867.070040] LustreError: 164592:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 4872.159938] LustreError: 163900:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a4f0cfea x1875367312782976/t0(0) o250->MGC192.168.203.137@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4872.377357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4872.465938] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4875.970890] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4877.685886] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4877.695925] Lustre: lustre-MDT0000: Denying connection for new client d83231e2-9ff8-44b6-939f-beb842c7682b (at 192.168.203.37@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 4877.800662] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4882.838759] Lustre: lustre-MDT0000: Denying connection for new client d83231e2-9ff8-44b6-939f-beb842c7682b (at 192.168.203.37@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [ 4888.038203] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4888.041537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4888.047167] Lustre: Skipped 2 previous similar messages [ 4888.062093] Lustre: Skipped 1 previous similar message [ 4891.364850] Lustre: server umount lustre-OST0000 complete [ 4894.741684] LustreError: 168323:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4894.746620] Lustre: 167671:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4894.754694] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 4894.771915] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 4894.778826] LustreError: 167671:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000059b82bd2 x1875367312797568/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4894.789818] LustreError: 167671:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 4894.797315] LustreError: 167671:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 4894.804852] Lustre: lustre-MDT0000: Recovery over after 0:17, of 2 clients 0 recovered and 2 were evicted. [ 4895.162215] Lustre: server umount lustre-MDT0000 complete [ 4898.283384] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4898.320599] LustreError: 164587:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4898.357106] LustreError: 164587:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4899.357947] LustreError: 164574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788489762 with bad export cookie 16426468222894798281 [ 4899.368313] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4899.379405] LustreError: 164574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4901.853437] Lustre: server umount lustre-MDT0001 complete [ 4913.370430] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 4916.305882] Key type lgssc unregistered [ 4916.693693] LNet: 169074:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4916.700700] LNetError: 169074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4916.720744] LNet: Removed LNI 192.168.203.137@tcp [ 4917.451644] Key type .llcrypt unregistered [ 4917.454666] Key type ._llcrypt unregistered [ 4935.852519] Key type ._llcrypt registered [ 4935.854496] Key type .llcrypt registered [ 4935.956986] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 4936.807217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4936.921049] alg: No test for adler32 (adler32-zlib) [ 4937.974364] Lustre: Lustre: Build Version: 2.17.57_84_g5b8c505 [ 4938.144992] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 4939.823145] Key type lgssc registered [ 4940.741868] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4947.376250] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4947.388650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4948.899898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4953.320164] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4959.201524] LustreError: 170460:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4959.224117] LustreError: 170460:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4962.047472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4962.484980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4966.745937] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4972.298958] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4977.004920] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4982.906556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4983.182434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4988.466753] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4993.461642] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 4994.856771] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5002.721150] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5002.731670] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5002.752459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5003.752504] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5003.756077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5003.791572] Lustre: Skipped 1 previous similar message [ 5005.990460] Lustre: server umount lustre-OST0000 complete [ 5013.473801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5013.483167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5013.505746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5015.431154] Lustre: server umount lustre-MDT0000 complete [ 5018.851989] LustreError: 170441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788489881 with bad export cookie 4316136484828297494 [ 5018.859113] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5018.860519] LustreError: 170441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5019.271406] Lustre: server umount lustre-MDT0001 complete [ 5028.192774] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 5030.501641] Key type lgssc unregistered [ 5030.729132] LNet: 173992:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5030.752179] LNetError: 173992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5030.775053] LNet: Removed LNI 192.168.203.137@tcp [ 5031.597641] Key type .llcrypt unregistered [ 5031.599827] Key type ._llcrypt unregistered [ 5065.850184] Key type ._llcrypt registered [ 5065.852264] Key type .llcrypt registered [ 5065.934290] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 5066.689063] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5066.747525] alg: No test for adler32 (adler32-zlib) [ 5067.845857] Lustre: Lustre: Build Version: 2.17.57_84_g5b8c505 [ 5068.059175] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 5069.767298] Key type lgssc registered [ 5070.669034] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5076.890636] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5076.905244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5078.255549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5081.636500] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5087.712944] LustreError: 175700:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5087.731184] LustreError: 175700:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5088.734726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5089.340769] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5094.520197] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5103.581559] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5110.080026] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5116.530721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5116.773888] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5122.240431] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5122.926913] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5130.084513] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5143.521955] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5143.539185] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5143.559680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5144.545249] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5144.553640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5144.563668] LustreError: Skipped 1 previous similar message [ 5144.588520] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5149.678469] LustreError: 177523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5149.786788] Lustre: server umount lustre-OST0000 complete [ 5154.796867] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5154.818420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5154.829623] Lustre: Skipped 2 previous similar messages [ 5159.905199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5159.929588] Lustre: Skipped 1 previous similar message [ 5160.321722] Lustre: server umount lustre-MDT0000 complete [ 5165.030188] LustreError: 177348:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5165.058929] LustreError: 177348:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5165.134204] LustreError: 176433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788490027 with bad export cookie 14027990281729584851 [ 5165.148644] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5165.155758] LustreError: 176433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5165.453457] Lustre: server umount lustre-MDT0001 complete [ 5176.267750] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 5179.138661] Key type lgssc unregistered [ 5179.506409] LNet: 179249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5179.524711] LNetError: 179249:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5179.564462] LNet: Removed LNI 192.168.203.137@tcp [ 5180.930776] Key type .llcrypt unregistered [ 5180.934928] Key type ._llcrypt unregistered [ 5205.723438] Key type ._llcrypt registered [ 5205.734977] Key type .llcrypt registered [ 5205.916652] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 5207.244238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5207.448035] alg: No test for adler32 (adler32-zlib) [ 5208.457798] Lustre: Lustre: Build Version: 2.17.57_84_g5b8c505 [ 5208.601099] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 5210.207238] Key type lgssc registered [ 5211.082296] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5219.754691] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5219.770968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5221.092192] Lustre: *** cfs_fail_loc=135, val=0*** [ 5221.151190] LustreError: 180768:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5221.153719] LustreError: 180768:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.137@tcp: cfg command failed: rc = -2 [ 5221.161777] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5221.161777] [ 5221.171253] LustreError: MGC192.168.203.137@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5221.180752] LustreError: 180729:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5221.185770] LustreError: 180729:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 5221.190389] LustreError: 180729:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 5221.194503] LustreError: 180729:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5221.234466] Lustre: server umount lustre-MDT0000 complete [ 5221.238620] LustreError: 180729:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5226.643557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5227.066406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5230.071381] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5237.217491] LustreError: 181140:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5237.236032] LustreError: 181140:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5239.501371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5240.214403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5245.002082] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5252.621188] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5258.072522] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5260.769231] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5260.779234] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5260.798285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5261.792765] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5261.795552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5261.822016] Lustre: Skipped 1 previous similar message [ 5264.929696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5264.937026] Lustre: Skipped 1 previous similar message [ 5266.562959] Lustre: server umount lustre-MDT0000 complete [ 5269.986392] LustreError: 181141:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5270.031442] LustreError: 181141:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5270.881420] LustreError: 181121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788490133 with bad export cookie 4233134673048417059 [ 5270.899093] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5270.901141] LustreError: 181121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5271.614094] Lustre: server umount lustre-MDT0001 complete [ 5282.154733] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 5285.227353] Key type lgssc unregistered [ 5285.632159] LNet: 183461:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5285.640151] LNetError: 183461:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5285.659115] LNet: Removed LNI 192.168.203.137@tcp [ 5286.531666] Key type .llcrypt unregistered [ 5286.533460] Key type ._llcrypt unregistered [ 5304.499468] Key type ._llcrypt registered [ 5304.501881] Key type .llcrypt registered [ 5304.609118] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 5304.875571] Key type .llcrypt unregistered [ 5304.878077] Key type ._llcrypt unregistered [ 5319.746594] Key type ._llcrypt registered [ 5319.749852] Key type .llcrypt registered [ 5319.861182] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 5320.787534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5320.836837] alg: No test for adler32 (adler32-zlib) [ 5321.846849] Lustre: Lustre: Build Version: 2.17.57_84_g5b8c505 [ 5322.057473] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 5323.743210] Key type lgssc registered [ 5324.794690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5331.522326] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5331.538850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5347.104253] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5352.224693] LustreError: 185583:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.137@tcp: failed processing log, type 4: rc = -110 [ 5378.847882] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5385.085040] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5391.467707] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5397.415883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5398.175457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5398.264017] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5401.985749] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5403.619796] LustreError: 185609:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5403.653734] LustreError: 185609:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5408.744418] LustreError: 185608:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5412.402629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5412.774456] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5418.670499] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5426.941225] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5432.900695] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5435.945984] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5448.674027] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5448.683983] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5448.707671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5449.703864] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5449.705530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5449.728658] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5449.765769] Lustre: Skipped 1 previous similar message [ 5454.309792] Lustre: server umount lustre-OST0000 complete [ 5458.912174] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5458.927360] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5458.945016] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5465.061119] LustreError: 188616:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5465.078602] LustreError: 188616:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5465.213785] Lustre: server umount lustre-MDT0000 complete [ 5469.046991] LustreError: 185589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788490331 with bad export cookie 3460860861409225172 [ 5469.060877] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5469.073811] LustreError: 185589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5469.314572] Lustre: server umount lustre-MDT0001 complete [ 5478.122679] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 5481.086672] Key type lgssc unregistered [ 5481.363646] LNet: 189548:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5481.376258] LNetError: 189548:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5481.405247] LNet: Removed LNI 192.168.203.137@tcp [ 5482.381993] Key type .llcrypt unregistered [ 5482.386732] Key type ._llcrypt unregistered [ 5501.275304] Key type ._llcrypt registered [ 5501.277434] Key type .llcrypt registered [ 5501.375552] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 5502.133470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5502.241216] alg: No test for adler32 (adler32-zlib) [ 5503.276318] Lustre: Lustre: Build Version: 2.17.57_84_g5b8c505 [ 5503.466191] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 5505.122291] Key type lgssc registered [ 5506.189657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5512.479394] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5512.499444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5513.822051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5517.238174] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5523.428624] LustreError: 190938:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5523.448710] LustreError: 190938:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5525.313064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5525.893613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5532.465870] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5541.172244] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5547.242111] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5553.371773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5553.730359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5558.826977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5559.176233] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5565.897381] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5571.490983] Lustre: Failing over lustre-MDT0000 [ 5571.839218] Lustre: server umount lustre-MDT0000 complete [ 5572.064783] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5572.079380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5572.107175] LustreError: 190939:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5573.156059] LustreError: 190938:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5573.169859] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5573.193603] Lustre: Skipped 1 previous similar message [ 5578.211148] LustreError: 191864:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5578.223403] LustreError: 191864:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 5583.329919] LustreError: 190939:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5583.369471] LustreError: 190939:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5588.447589] Lustre: 190250:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788490435/real 1788490435] req@000000005ff8c90e x1875368074043136/t0(0) o400->MGC192.168.203.137@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788490451 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5588.471669] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5591.209292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5593.570951] LustreError: 190935:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5593.593708] LustreError: 190935:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 5598.700747] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x72d805f5503903d1 [ 5598.717716] LustreError: 194000:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.203.137@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (00000000776c6e83) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 5598.729447] Lustre: MGC192.168.203.137@tcp: Connection restored to 0@lo (at 0@lo) [ 5598.952706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5599.045943] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5600.148544] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5604.348773] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5604.383676] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5604.430852] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5604.785402] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5609.822833] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5617.805592] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5624.802175] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5624.808634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5624.819725] Lustre: Skipped 1 previous similar message [ 5626.896460] Lustre: server umount lustre-OST0000 complete [ 5635.040405] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5635.045169] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5635.047453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5635.047462] Lustre: Skipped 1 previous similar message [ 5635.060569] Lustre: Skipped 1 previous similar message [ 5640.161154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5640.172697] Lustre: Skipped 2 previous similar messages [ 5645.280311] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5645.281809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5645.296693] Lustre: Skipped 1 previous similar message [ 5645.468752] Lustre: server umount lustre-MDT0000 complete [ 5648.784371] LustreError: 190918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788490511 with bad export cookie 8275370866464326609 [ 5648.787638] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5648.794822] LustreError: 190918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5649.117247] Lustre: server umount lustre-MDT0001 complete [ 5659.585961] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 5662.766207] Key type lgssc unregistered [ 5663.056414] LNet: 195827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5663.065151] LNetError: 195827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5663.093199] LNet: Removed LNI 192.168.203.137@tcp [ 5664.028803] Key type .llcrypt unregistered [ 5664.030755] Key type ._llcrypt unregistered [ 5686.883836] Key type ._llcrypt registered [ 5686.887959] Key type .llcrypt registered [ 5687.055705] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 5688.234888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5688.341405] alg: No test for adler32 (adler32-zlib) [ 5689.354319] Lustre: Lustre: Build Version: 2.17.57_84_g5b8c505 [ 5689.583366] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 5691.247492] Key type lgssc registered [ 5692.249893] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5699.764889] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5699.787113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5701.217675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5705.297484] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5710.305725] LustreError: 197234:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5710.324690] LustreError: 197234:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5712.339293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5712.631504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5716.254261] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5724.228583] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5729.690955] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5735.576975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5735.969269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5742.136029] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5746.242465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5749.579659] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5753.448399] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5758.845913] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5769.700865] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5776.876859] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5776.898186] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5776.902573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5776.913892] Lustre: Skipped 1 previous similar message [ 5781.469770] Lustre: server umount lustre-OST0000 complete [ 5787.104371] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5787.117429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5787.123035] Lustre: Skipped 2 previous similar messages [ 5787.136197] Lustre: Skipped 1 previous similar message [ 5792.227463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5796.324372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5796.332354] Lustre: Skipped 2 previous similar messages [ 5799.391192] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5799.558484] Lustre: server umount lustre-MDT0000 complete [ 5801.441215] LustreError: 197235:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5801.458499] LustreError: 197235:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5802.833787] LustreError: 197215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788490665 with bad export cookie 11457088111366216462 [ 5802.836794] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5802.850483] LustreError: 197215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5803.169760] Lustre: server umount lustre-MDT0001 complete [ 5812.939254] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 5815.583308] Key type lgssc unregistered [ 5815.917480] LNet: 201077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5815.932457] LNetError: 201077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5815.957830] LNet: Removed LNI 192.168.203.137@tcp [ 5816.875935] Key type .llcrypt unregistered [ 5816.879538] Key type ._llcrypt unregistered [ 5833.299831] Key type ._llcrypt registered [ 5833.302306] Key type .llcrypt registered [ 5833.395115] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 5834.180213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5834.269753] alg: No test for adler32 (adler32-zlib) [ 5835.274789] Lustre: Lustre: Build Version: 2.17.57_84_g5b8c505 [ 5835.464968] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 5837.096426] Key type lgssc registered [ 5837.840918] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5843.369995] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5843.390283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5844.859768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5848.016155] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5854.117891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5854.411054] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5858.255096] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5866.767927] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5874.656062] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5881.220901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5881.529324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5886.895982] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5890.735678] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 5892.819788] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5897.657604] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 5908.184469] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5918.813720] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 5925.559734] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 5936.101484] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5936.119346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5936.123615] Lustre: Skipped 1 previous similar message [ 5939.933731] Lustre: server umount lustre-OST0000 complete [ 5946.337361] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5946.342729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5946.348757] Lustre: Skipped 2 previous similar messages [ 5946.355725] Lustre: Skipped 1 previous similar message [ 5949.233552] Lustre: server umount lustre-MDT0000 complete [ 5951.457304] LustreError: 202337:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5951.466600] LustreError: 202337:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5952.292842] LustreError: 205012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788490815 with bad export cookie 13716784995945807738 [ 5952.293270] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5952.302798] LustreError: 205012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5952.683524] Lustre: server umount lustre-MDT0001 complete [ 5961.618628] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 5963.954354] Key type lgssc unregistered [ 5964.202637] LNet: 207354:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5964.217811] LNetError: 207354:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5965.289232] LNet: Removed LNI 192.168.203.137@tcp [ 5966.088175] Key type .llcrypt unregistered [ 5966.091879] Key type ._llcrypt unregistered [ 5985.932286] Key type ._llcrypt registered [ 5985.935411] Key type .llcrypt registered [ 5986.033435] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 5986.743986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5986.803146] alg: No test for adler32 (adler32-zlib) [ 5987.877549] Lustre: Lustre: Build Version: 2.17.57_84_g5b8c505 [ 5988.112360] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 5989.767161] Key type lgssc registered [ 5990.749643] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5997.116149] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5997.130611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5998.607925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6002.353194] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6008.802380] LustreError: 208742:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6008.823944] LustreError: 208742:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6008.860441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6009.108643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6012.949388] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6018.309808] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6022.176869] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6027.484730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6027.762424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6032.742408] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6039.192447] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6044.150539] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6059.487981] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6059.494949] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6059.498494] LustreError: Skipped 1 previous similar message [ 6059.500634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6059.539211] Lustre: Skipped 1 previous similar message [ 6064.613866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6064.626532] Lustre: Skipped 1 previous similar message [ 6065.311660] Lustre: server umount lustre-OST0000 complete [ 6069.728268] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6069.731926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6069.737073] Lustre: Skipped 1 previous similar message [ 6069.744110] Lustre: Skipped 2 previous similar messages [ 6074.646629] Lustre: server umount lustre-MDT0000 complete [ 6074.851087] LustreError: 211245:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6074.874967] LustreError: 211245:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6077.904987] LustreError: 208722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788490940 with bad export cookie 6979016392531551681 [ 6077.917872] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6077.924493] LustreError: 208722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6078.377040] Lustre: server umount lustre-MDT0001 complete [ 6089.734949] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 6092.465838] Key type lgssc unregistered [ 6092.755475] LNet: 212454:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6092.760064] LNetError: 212454:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6092.785721] LNet: Removed LNI 192.168.203.137@tcp [ 6093.549310] Key type .llcrypt unregistered [ 6093.551839] Key type ._llcrypt unregistered [ 6111.089149] Key type ._llcrypt registered [ 6111.092685] Key type .llcrypt registered [ 6111.177803] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 6111.916118] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6111.983262] alg: No test for adler32 (adler32-zlib) [ 6113.025806] Lustre: Lustre: Build Version: 2.17.57_84_g5b8c505 [ 6113.218786] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 6114.871197] Key type lgssc registered [ 6116.388129] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6124.935577] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6124.957223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6126.285116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6129.771431] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6135.777894] LustreError: 213715:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6135.791596] LustreError: 213715:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6138.552100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6139.091643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6143.577057] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6150.903422] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6157.438640] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6164.513651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6165.019735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6171.210935] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6171.504699] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6178.650983] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6193.632238] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6193.636601] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6193.647168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6193.664489] Lustre: Skipped 1 previous similar message [ 6198.757855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6198.766911] Lustre: Skipped 1 previous similar message [ 6199.687449] Lustre: server umount lustre-OST0000 complete [ 6203.872601] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6203.876095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6203.887474] Lustre: Skipped 1 previous similar message [ 6203.911192] Lustre: Skipped 2 previous similar messages [ 6208.992710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6209.006919] Lustre: Skipped 1 previous similar message [ 6209.713910] Lustre: server umount lustre-MDT0000 complete [ 6213.546090] LustreError: 213697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788491076 with bad export cookie 14457292524574380629 [ 6213.549930] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6213.564527] LustreError: 213697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6213.882887] Lustre: server umount lustre-MDT0001 complete [ 6223.577905] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 6226.116681] Key type lgssc unregistered [ 6226.393202] LNet: 217397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6226.402182] LNetError: 217397:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6226.432897] LNet: Removed LNI 192.168.203.137@tcp [ 6227.304351] Key type .llcrypt unregistered [ 6227.308819] Key type ._llcrypt unregistered [ 6245.179928] Key type ._llcrypt registered [ 6245.182445] Key type .llcrypt registered [ 6245.283104] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 6246.544199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6246.564279] alg: No test for adler32 (adler32-zlib) [ 6247.753301] Lustre: Lustre: Build Version: 2.17.57_84_g5b8c505 [ 6248.080346] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 6249.799176] Key type lgssc registered [ 6250.882316] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6258.264332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6258.281737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6259.639020] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6262.919674] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6269.928322] LustreError: 218659:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6269.954397] LustreError: 218659:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6270.264447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6270.681958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6275.133649] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6283.357504] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6289.397806] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6296.256893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6296.619122] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6303.075864] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6305.841912] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6309.988747] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6319.012615] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6319.198464] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6319.546717] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6327.175479] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6334.922288] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6336.330329] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6348.222428] Lustre: Permanently deactivating lustre-OST0001 [ 6348.239634] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6356.572897] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6366.387614] Lustre: Permanently reactivating lustre-OST0001 [ 6366.414766] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6366.420343] Lustre: Skipped 2 previous similar messages [ 6366.507460] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6366.543318] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6366.567404] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6366.590896] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6374.405251] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6374.422510] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6376.172608] Lustre: server umount lustre-OST0001 complete [ 6379.504176] LustreError: 220481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6379.530487] LustreError: 220481:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6382.562336] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6382.567422] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6382.573187] LustreError: Skipped 1 previous similar message [ 6382.584753] Lustre: Skipped 1 previous similar message [ 6382.591616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6382.604027] Lustre: Skipped 2 previous similar messages [ 6384.610227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6384.612214] LustreError: 220481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6384.621257] Lustre: Skipped 1 previous similar message [ 6384.652478] LustreError: 220481:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6385.861229] Lustre: server umount lustre-OST0000 complete [ 6389.728889] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6389.744057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6394.847162] LustreError: 222595:0:(osp_dev.c:699:osp_process_config()) lustre-OST0001-osc-MDT0001: unknown param osc.active=1 [ 6394.851624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6394.871086] Lustre: Skipped 2 previous similar messages [ 6395.915748] Lustre: server umount lustre-MDT0000 complete [ 6399.497693] LustreError: 221497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788491262 with bad export cookie 17976647734102280603 [ 6399.498973] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6399.514350] LustreError: 221497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6400.110298] Lustre: server umount lustre-MDT0001 complete [ 6409.466356] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 6412.172913] Key type lgssc unregistered [ 6412.544506] LNet: 223748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6412.554480] LNetError: 223748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6412.568263] LNet: Removed LNI 192.168.203.137@tcp [ 6413.606753] Key type .llcrypt unregistered [ 6413.611286] Key type ._llcrypt unregistered [ 6451.605081] Key type ._llcrypt registered [ 6451.607466] Key type .llcrypt registered [ 6451.739432] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 6452.567112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6452.659237] alg: No test for adler32 (adler32-zlib) [ 6453.837263] Lustre: Lustre: Build Version: 2.17.57_84_g5b8c505 [ 6454.079598] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 6455.719159] Key type lgssc registered [ 6456.741397] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6463.600431] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6463.651427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6464.948588] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6464.966470] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6465.124180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6469.084496] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6475.882139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6475.988650] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6476.041146] Lustre: 225950:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6476.256379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6479.008462] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6485.246329] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6491.155143] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6496.875180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6497.103686] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6497.426065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6503.443854] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6504.550767] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6509.579506] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6513.078208] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6516.262739] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6522.666453] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6529.445906] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6542.882520] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6542.889111] Lustre: Skipped 1 previous similar message [ 6560.528785] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6560.541820] Lustre: Skipped 1 previous similar message [ 6595.556652] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6595.567264] Lustre: Skipped 3 previous similar messages [ 6659.786396] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6659.804795] Lustre: Skipped 8 previous similar messages [ 6678.499905] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6678.520509] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6678.568899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6679.008944] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6684.136414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6684.145897] Lustre: Skipped 2 previous similar messages [ 6689.251112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6689.263826] Lustre: Skipped 1 previous similar message [ 6690.271186] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6690.437922] Lustre: server umount lustre-OST0000 complete [ 6694.369372] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6694.382589] Lustre: Skipped 1 previous similar message [ 6694.390040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6694.397455] Lustre: Skipped 1 previous similar message [ 6699.489803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6699.500692] Lustre: Skipped 1 previous similar message [ 6700.639063] Lustre: server umount lustre-MDT0000 complete [ 6704.639258] LustreError: 225941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788491567 with bad export cookie 2965058296293776728 [ 6704.653973] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6704.656956] LustreError: 225941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6705.026831] Lustre: server umount lustre-MDT0001 complete [ 6715.330744] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 6718.225175] Key type lgssc unregistered [ 6718.501692] LNet: 230736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6718.515440] LNetError: 230736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6719.599689] LNet: Removed LNI 192.168.203.137@tcp [ 6720.497419] Key type .llcrypt unregistered [ 6720.500716] Key type ._llcrypt unregistered [ 6741.581175] Key type ._llcrypt registered [ 6741.583581] Key type .llcrypt registered [ 6741.693730] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 6742.464731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6742.478490] alg: No test for adler32 (adler32-zlib) [ 6743.548566] Lustre: Lustre: Build Version: 2.17.57_84_g5b8c505 [ 6743.824644] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 6745.520762] Key type lgssc registered [ 6746.627327] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6754.220646] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6754.252705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6755.722672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6759.548341] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6766.049154] LustreError: 232125:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6766.074287] LustreError: 232125:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6766.950506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6767.318712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6771.116138] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6777.915629] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6783.441306] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6789.758498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6790.095893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6795.767891] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6800.384445] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 6802.207662] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6822.883581] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6822.888936] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6822.917028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6823.562100] Lustre: server umount lustre-OST0000 complete [ 6829.026952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6829.035406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6829.048278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6831.073207] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6831.074835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6831.099703] Lustre: Skipped 1 previous similar message [ 6836.194711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6836.207544] Lustre: Skipped 1 previous similar message [ 6838.460940] Lustre: server umount lustre-MDT0000 complete [ 6840.293527] LustreError: 232124:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6840.331760] LustreError: 232124:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6842.313523] LustreError: 232104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788491705 with bad export cookie 16982086940087600848 [ 6842.325732] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6842.325779] LustreError: 232104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6842.746774] Lustre: server umount lustre-MDT0001 complete [ 6854.012559] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 6856.979424] Key type lgssc unregistered [ 6857.322292] LNet: 235817:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6857.332564] LNetError: 235817:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6857.363408] LNet: Removed LNI 192.168.203.137@tcp [ 6858.327695] Key type .llcrypt unregistered [ 6858.333066] Key type ._llcrypt unregistered [ 6879.110300] Key type ._llcrypt registered [ 6879.112111] Key type .llcrypt registered [ 6879.205861] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 6879.796511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6879.871661] alg: No test for adler32 (adler32-zlib) [ 6880.925987] Lustre: Lustre: Build Version: 2.17.57_84_g5b8c505 [ 6881.129947] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 6882.767164] Key type lgssc registered [ 6884.038145] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7159.036688] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 7161.915340] Key type lgssc unregistered [ 7162.294662] LNet: 237817:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7162.310415] LNetError: 237817:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7162.338375] LNet: Removed LNI 192.168.203.137@tcp [ 7163.191550] Key type .llcrypt unregistered [ 7163.193187] Key type ._llcrypt unregistered [ 7188.834653] Key type ._llcrypt registered [ 7188.837409] Key type .llcrypt registered [ 7188.956561] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 7203.721969] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 7204.650427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7204.715763] alg: No test for adler32 (adler32-zlib) [ 7205.771484] Lustre: Lustre: Build Version: 2.17.57_84_g5b8c505 [ 7206.031332] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 7207.727867] Key type lgssc registered [ 7208.924446] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7236.490635] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6967 sec ========== 23:21:36 (1788492096) [ 7238.042495] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:21:38 (1788492098) === [ 7241.130578] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:21:41 (1788492101) === [ 7280.177603] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 7282.899289] Key type lgssc unregistered [ 7283.200655] LNet: 243044:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7283.209498] LNetError: 243044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7283.242973] LNet: Removed LNI 192.168.203.137@tcp [ 7283.991703] Key type .llcrypt unregistered [ 7283.995634] Key type ._llcrypt unregistered