[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 729792202 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a2000-0x1465ccfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002376] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.004018] kvm-guest: setup PV IPIs [ 0.006606] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007028] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008016] pid_max: default: 32768 minimum: 301 [ 0.009140] LSM: Security Framework initializing [ 0.010065] Yama: becoming mindful. [ 0.011044] SELinux: Initializing. [ 0.012067] *** VALIDATE selinux *** [ 0.020685] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025448] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026194] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027129] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028133] *** VALIDATE tmpfs *** [ 0.030565] *** VALIDATE proc *** [ 0.032003] *** VALIDATE cgroup *** [ 0.033015] *** VALIDATE cgroup2 *** [ 0.035083] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036177] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038033] Spectre V2 : User space: Vulnerable [ 0.039016] Speculative Store Bypass: Vulnerable [ 0.042530] debug: unmapping init [mem 0xffffffff9ac59000-0xffffffff9ac60fff] [ 0.044952] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045622] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046030] ... version: 2 [ 0.047016] ... bit width: 48 [ 0.048015] ... generic registers: 4 [ 0.049016] ... value mask: 0000ffffffffffff [ 0.050023] ... max period: 00007fffffffffff [ 0.051016] ... fixed-purpose events: 3 [ 0.052014] ... event mask: 000000070000000f [ 0.053311] rcu: Hierarchical SRCU implementation. [ 0.055513] smp: Bringing up secondary CPUs ... [ 0.056641] x86: Booting SMP configuration: [ 0.057031] .... node #0, CPUs: #1 #2 #3 [ 0.060561] smp: Brought up 1 node, 4 CPUs [ 0.062016] smpboot: Max logical packages: 1 [ 0.063021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.222255] node 0 deferred pages initialised in 155ms [ 0.225029] devtmpfs: initialized [ 0.226223] x86/mm: Memory block size: 128MB [ 0.228725] gcov: version magic: 0x41383552 [ 0.229646] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.233095] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.235367] pinctrl core: initialized pinctrl subsystem [ 0.237209] [ 0.237775] ************************************************************* [ 0.240016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.242011] ** ** [ 0.244015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.247016] ** ** [ 0.249011] ** This means that this kernel is built to expose internal ** [ 0.250009] ** IOMMU data structures, which may compromise security on ** [ 0.252013] ** your system. ** [ 0.254014] ** ** [ 0.256015] ** If you see this message and you are not debugging the ** [ 0.258014] ** kernel, report this immediately to your vendor! ** [ 0.260015] ** ** [ 0.262014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.264014] ************************************************************* [ 0.266720] NET: Registered protocol family 16 [ 0.268438] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.270067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.273069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.276034] cpuidle: using governor menu [ 0.277746] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.280084] PCI: Using configuration type 1 for base access [ 0.282154] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.291143] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.292049] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.295045] cryptd: max_cpu_qlen set to 1000 [ 0.299325] ACPI: Added _OSI(Module Device) [ 0.301021] ACPI: Added _OSI(Processor Device) [ 0.303015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.304013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.309157] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.315561] ACPI: Interpreter enabled [ 0.317072] ACPI: PM: (supports S0 S3 S4 S5) [ 0.318015] ACPI: Using IOAPIC for interrupt routing [ 0.320130] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.325190] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.335217] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.337052] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.340026] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.343072] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.348000] acpiphp: Slot [2] registered [ 0.349115] acpiphp: Slot [5] registered [ 0.350107] acpiphp: Slot [6] registered [ 0.352115] acpiphp: Slot [7] registered [ 0.353097] acpiphp: Slot [8] registered [ 0.355142] acpiphp: Slot [9] registered [ 0.357092] acpiphp: Slot [10] registered [ 0.358143] acpiphp: Slot [3] registered [ 0.360096] acpiphp: Slot [4] registered [ 0.361137] acpiphp: Slot [11] registered [ 0.362115] acpiphp: Slot [12] registered [ 0.364094] acpiphp: Slot [13] registered [ 0.365119] acpiphp: Slot [14] registered [ 0.366118] acpiphp: Slot [15] registered [ 0.368117] acpiphp: Slot [16] registered [ 0.369091] acpiphp: Slot [17] registered [ 0.371131] acpiphp: Slot [18] registered [ 0.372113] acpiphp: Slot [19] registered [ 0.374101] acpiphp: Slot [20] registered [ 0.375053] acpiphp: Slot [21] registered [ 0.376161] acpiphp: Slot [22] registered [ 0.378118] acpiphp: Slot [23] registered [ 0.379083] acpiphp: Slot [24] registered [ 0.380082] acpiphp: Slot [25] registered [ 0.381078] acpiphp: Slot [26] registered [ 0.382093] acpiphp: Slot [27] registered [ 0.384131] acpiphp: Slot [28] registered [ 0.385125] acpiphp: Slot [29] registered [ 0.386114] acpiphp: Slot [30] registered [ 0.388097] acpiphp: Slot [31] registered [ 0.390074] PCI host bridge to bus 0000:00 [ 0.391022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.394026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.396027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.399030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.401035] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.404036] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.406180] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.409031] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.412012] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.420647] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.426055] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.429026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.430014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.433029] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.436160] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.439750] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.442050] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.445951] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.451020] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.464021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.469913] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.474687] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.481017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.490019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.506022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.516121] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.526018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.532021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.550029] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.561481] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.568018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.579019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.602018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.612242] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.618827] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.624019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.639027] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.651680] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.656985] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.662029] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.677022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.688284] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.696016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.702024] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.719024] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.728658] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.732420] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.734372] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.736299] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.738515] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.742200] iommu: Default domain type: Passthrough [ 0.744573] SCSI subsystem initialized [ 0.746146] ACPI: bus type USB registered [ 0.747137] usbcore: registered new interface driver usbfs [ 0.750103] usbcore: registered new interface driver hub [ 0.752094] usbcore: registered new device driver usb [ 0.754225] pps_core: LinuxPPS API ver. 1 registered [ 0.756013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.759078] PTP clock support registered [ 0.761166] EDAC MC: Ver: 3.0.0 [ 0.764211] PCI: Using ACPI for IRQ routing [ 0.765483] NetLabel: Initializing [ 0.767018] NetLabel: domain hash size = 128 [ 0.768010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.770098] NetLabel: unlabeled traffic allowed by default [ 0.772164] vgaarb: loaded [ 0.773303] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.775015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.779331] clocksource: Switched to clocksource kvm-clock [ 0.892587] VFS: Disk quotas dquot_6.6.0 [ 0.894660] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.898900] *** VALIDATE ramfs *** [ 0.900279] *** VALIDATE hugetlbfs *** [ 0.901865] pnp: PnP ACPI init [ 0.904303] pnp: PnP ACPI: found 6 devices [ 0.923541] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.926654] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.928717] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.930744] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.933129] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.935537] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.938391] NET: Registered protocol family 2 [ 0.940893] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.945779] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.948846] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.954072] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.957154] TCP: Hash tables configured (established 65536 bind 65536) [ 0.959568] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.962597] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.965373] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.968501] NET: Registered protocol family 1 [ 0.970932] RPC: Registered named UNIX socket transport module. [ 0.973300] RPC: Registered udp transport module. [ 0.975323] RPC: Registered tcp transport module. [ 0.977347] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.979760] NET: Registered protocol family 44 [ 0.981388] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.982930] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.984931] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.987293] PCI: CLS 0 bytes, default 64 [ 0.988718] Unpacking initramfs... [ 2.404638] debug: unmapping init [mem 0xffff97ffbcc54000-0xffff97ffbffbffff] [ 2.408714] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.411219] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.414024] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.932782] Initialise system trusted keyrings [ 2.934659] Key type blacklist registered [ 2.936536] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.945204] zbud: loaded [ 2.948158] *** VALIDATE nfs *** [ 2.949520] *** VALIDATE nfs4 *** [ 2.951304] pstore: using deflate compression [ 2.955223] Platform Keyring initialized [ 3.056363] NET: Registered protocol family 38 [ 3.058107] Key type asymmetric registered [ 3.059544] Asymmetric key parser 'x509' registered [ 3.061497] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.064621] io scheduler mq-deadline registered [ 3.066094] io scheduler kyber registered [ 3.067608] io scheduler bfq registered [ 3.070350] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.073524] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.076170] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.078439] ACPI: Power Button [PWRF] [ 3.084322] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.092188] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.108321] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.122916] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.135564] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.165842] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.196216] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.201789] Non-volatile memory driver v1.3 [ 3.203315] Linux agpgart interface v0.103 [ 3.234632] virtio_blk virtio1: [vda] 145792 512-byte logical blocks (74.6 MB/71.2 MiB) [ 3.237926] vda: detected capacity change from 0 to 74645504 [ 3.252790] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.255213] vdb: detected capacity change from 0 to 1073741824 [ 3.270308] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.273284] vdc: detected capacity change from 0 to 2621440000 [ 3.287135] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.289528] vdd: detected capacity change from 0 to 2621440000 [ 3.302610] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.305160] vde: detected capacity change from 0 to 4294967296 [ 3.318104] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.321208] vdf: detected capacity change from 0 to 4294967296 [ 3.327713] libphy: Fixed MDIO Bus: probed [ 3.336932] usbcore: registered new interface driver usbserial_generic [ 3.338916] usbserial: USB Serial support registered for generic [ 3.341221] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.345153] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.347049] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.349643] mousedev: PS/2 mouse device common for all mice [ 3.353863] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.354437] rtc_cmos 00:05: RTC can wake from S4 [ 3.360469] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.360816] rtc_cmos 00:05: registered as rtc0 [ 3.364991] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.365765] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.367054] intel_pstate: CPU model not supported [ 3.372821] hid: raw HID events driver (C) Jiri Kosina [ 3.374422] usbcore: registered new interface driver usbhid [ 3.375810] usbhid: USB HID core driver [ 3.377346] drop_monitor: Initializing network drop monitor service [ 3.379063] Initializing XFRM netlink socket [ 3.380593] NET: Registered protocol family 10 [ 3.383061] Segment Routing with IPv6 [ 3.384644] NET: Registered protocol family 17 [ 3.385925] mpls_gso: MPLS GSO support [ 3.391933] RAS: Correctable Errors collector initialized. [ 3.394219] AVX version of gcm_enc/dec engaged. [ 3.395892] AES CTR mode by8 optimization enabled [ 3.473545] sched_clock: Marking stable (3473516102, 0)->(4361650920, -888134818) [ 3.477474] registered taskstats version 1 [ 3.479533] Loading compiled-in X.509 certificates [ 3.481811] zswap: loaded using pool lzo/zbud [ 3.507643] Key type big_key registered [ 3.519060] Key type encrypted registered [ 3.520516] ima: No TPM chip found, activating TPM-bypass! [ 3.522015] ima: Allocated hash algorithm: sha1 [ 3.523464] ima: No architecture policies found [ 3.525032] evm: Initialising EVM extended attributes: [ 3.526280] evm: security.selinux [ 3.527367] evm: security.ima [ 3.528189] evm: security.capability [ 3.529362] evm: HMAC attrs: 0x1 [ 3.532131] rtc_cmos 00:05: setting system clock to 2026-07-16 07:08:40 UTC (1784185720) [ 3.537882] debug: unmapping init [mem 0xffffffff9bc03000-0xffffffff9bdfffff] [ 3.540658] debug: unmapping init [mem 0xffffffff9a982000-0xffffffff9ac58fff] [ 3.550117] Write protecting the kernel read-only data: 28672k [ 3.553865] debug: unmapping init [mem 0xffffffff99003000-0xffffffff991fffff] [ 3.556727] debug: unmapping init [mem 0xffffffff99914000-0xffffffff999fffff] [ 3.592044] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.599439] systemd[1]: Detected virtualization kvm. [ 3.600619] systemd[1]: Detected architecture x86-64. [ 3.602722] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.625679] systemd[1]: No hostname configured. [ 3.627478] systemd[1]: Set hostname to . [ 3.630279] random: systemd: uninitialized urandom read (16 bytes read) [ 3.632840] systemd[1]: Initializing machine ID from random generator. [ 3.787327] random: systemd: uninitialized urandom read (16 bytes read) [ 3.789890] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.794368] random: systemd: uninitialized urandom read (16 bytes read) [ 3.797906] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.806590] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.474926] device-mapper: uevent: version 1.0.3 [ 4.477164] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.273987] virtio_net virtio0 ens2: renamed from eth0 [ 5.289450] random: fast init done [ 5.364799] scsi host0: ata_piix [ 5.378460] scsi host1: ata_piix [ 5.380833] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.383278] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.801402] dracut-initqueue[589]: RTNETLINK answers: File exists [ 10.058475] random: crng init done [ 10.060032] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.547259] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.024896] printk: systemd: 22 output lines suppressed due to ratelimiting [ 14.963094] SELinux: Disabled at runtime. [ 15.102094] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.125303] systemd[1]: Detected virtualization kvm. [ 15.130838] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 16.918324] systemd[1]: initrd-switch-root.service: Succeeded. [ 16.933554] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 16.963849] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 16.983893] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 16.993962] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 17.014062] systemd[1]: Starting Journal Service... Starting Journal Service... [ 17.032787] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ 17.090896] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 18.641082] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 19.858148] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 19.920158] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 20.549979] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 20.571891] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 26.606746] Key type dns_resolver registered [ 27.066950] NFS: Registering the id_resolver key type [ 27.068929] Key type id_resolver registered [ 27.070429] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg342-server login: [ 67.665126] hrtimer: interrupt took 3167117 ns [ 96.427962] libcfs: loading out-of-tree module taints kernel. [ 96.536988] Key type ._llcrypt registered [ 96.539216] Key type .llcrypt registered [ 96.672461] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 116.754795] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 118.587403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 118.602803] alg: No test for adler32 (adler32-zlib) [ 120.274976] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 121.152067] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 122.952196] Key type lgssc registered [ 124.829866] Lustre: Echo OBD driver; http://www.lustre.org/ [ 143.273280] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 186.013661] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 197.637857] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 197.681394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 198.950154] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 198.983207] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 199.060722] Lustre: lustre-MDT0000: new disk, initializing [ 199.135686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 199.156969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 202.866789] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 215.578175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 215.695351] Lustre: 6513:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 215.732420] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 215.735675] Lustre: Skipped 1 previous similar message [ 215.820477] Lustre: lustre-MDT0001: new disk, initializing [ 215.894853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 215.922652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 215.942151] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 220.387103] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 225.119052] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 234.814705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 235.201593] Lustre: lustre-OST0000: new disk, initializing [ 235.215863] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 235.221080] Lustre: 8452:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 235.312070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 240.127672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 240.133801] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 240.172751] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 240.708814] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 256.680164] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 256.825682] Lustre: lustre-OST0001: new disk, initializing [ 256.830342] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 256.836156] Lustre: 9525:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 256.906672] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 263.621681] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 265.268456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 265.276856] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 265.345959] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 276.080852] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 282.727911] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 289.778377] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing check_logdir /tmp/testlogs/ [ 295.470239] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing yml_node [ 300.355985] Lustre: DEBUG MARKER: Client: 2.17.54.162 [ 303.536521] Lustre: DEBUG MARKER: MDS: 2.17.54.162 [ 306.327995] Lustre: DEBUG MARKER: OSS: 2.17.54.162 [ 308.091532] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jul 16 03:13:42 EDT 2026 [ 326.215359] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 328.123193] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 347.107134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 347.109413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 347.137266] Lustre: Skipped 3 previous similar messages [ 347.153941] Lustre: Skipped 3 previous similar messages [ 351.318191] Lustre: server umount lustre-MDT0000 complete [ 355.206606] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784186072 with bad export cookie 6742356567819014886 [ 355.213040] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 355.218595] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 357.345537] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 357.347589] LustreError: 8947:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 357.347816] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 357.372149] Lustre: Skipped 1 previous similar message [ 357.412454] LustreError: 8947:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 360.353271] LustreError: 6521:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 360.353333] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 360.374849] LustreError: 6521:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 360.388038] Lustre: Skipped 2 previous similar messages [ 361.638684] Lustre: server umount lustre-MDT0001 complete [ 365.990770] Lustre: server umount lustre-OST0000 complete [ 370.316264] Lustre: server umount lustre-OST0001 complete [ 378.356638] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 390.586573] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 434.478496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 435.042943] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 435.115945] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 435.272214] Lustre: lustre-MDT0000: new disk, initializing [ 435.411416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 435.455893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 440.641923] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 451.693270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 451.863834] Lustre: 15080:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 451.959686] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 451.964641] Lustre: Skipped 1 previous similar message [ 452.058490] Lustre: lustre-MDT0001: new disk, initializing [ 452.166110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 452.212310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 452.228787] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 456.911436] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 466.335192] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 472.793295] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 479.645392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 480.000219] Lustre: lustre-OST0000: new disk, initializing [ 480.008902] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 480.025601] Lustre: 17045:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 480.164079] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 481.469316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 481.480586] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 481.669447] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 487.196553] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 498.022480] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 503.769704] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 504.042693] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 509.585817] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 509.805946] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 512.481337] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 512.483098] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 512.487414] LustreError: Skipped 1 previous similar message [ 512.499901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 512.510015] Lustre: Skipped 1 previous similar message [ 520.674507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 520.679670] Lustre: Skipped 3 previous similar messages [ 526.304216] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 526.660465] Lustre: server umount lustre-OST0000 complete [ 530.914830] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 530.920566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 530.933348] Lustre: Skipped 2 previous similar messages [ 530.945943] Lustre: Skipped 3 previous similar messages [ 545.248341] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 545.479470] Lustre: server umount lustre-MDT0000 complete [ 546.275845] LustreError: 15091:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 546.302461] LustreError: 15091:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 549.728067] LustreError: 17044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784186266 with bad export cookie 6742356567819016412 [ 549.729318] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 549.745588] LustreError: 17044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 550.081883] Lustre: server umount lustre-MDT0001 complete [ 553.907473] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 03:17:48 (1784186268) [ 562.048590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 562.628259] LustreError: 19316:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 562.712117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 567.506179] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 567.785698] LustreError: 19316:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 576.205679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 581.798496] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 589.991263] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 596.757913] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 604.155115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 611.760956] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 620.148751] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 631.780520] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 631.784972] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 631.792504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 631.796148] Lustre: Skipped 4 previous similar messages [ 637.425443] LustreError: 21136:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 637.445551] LustreError: 21136:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 637.562301] Lustre: server umount lustre-OST0000 complete [ 642.536743] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 642.561531] Lustre: Skipped 1 previous similar message [ 656.352386] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 656.767735] Lustre: server umount lustre-MDT0000 complete [ 657.889831] LustreError: 20044:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 657.908550] LustreError: 20044:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 661.319873] LustreError: 19298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784186378 with bad export cookie 6742356567819017301 [ 661.324799] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 661.325618] LustreError: 19298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 661.752612] Lustre: server umount lustre-MDT0001 complete [ 674.142312] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 676.884324] Key type lgssc unregistered [ 677.329435] LNet: 22854:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 677.352792] LNetError: 22854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 677.377312] LNet: Removed LNI 192.168.203.142@tcp [ 678.535174] Key type .llcrypt unregistered [ 678.537171] Key type ._llcrypt unregistered [ 701.143345] Key type ._llcrypt registered [ 701.145876] Key type .llcrypt registered [ 701.253771] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 702.293615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 702.367477] alg: No test for adler32 (adler32-zlib) [ 703.441211] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 703.674824] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 705.336239] Key type lgssc registered [ 706.496475] Lustre: Echo OBD driver; http://www.lustre.org/ [ 714.259556] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 714.280056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 715.703775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 720.337245] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 725.990580] LustreError: 24269:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 726.008230] LustreError: 24269:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 729.068250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 729.503276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 734.746322] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 743.164668] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 749.465314] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 756.283985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 756.645905] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 760.759760] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 763.072231] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 770.939966] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 786.401472] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 786.414247] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 786.430676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 788.348851] Lustre: server umount lustre-OST0000 complete [ 794.595125] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 794.596920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 794.622968] Lustre: Skipped 1 previous similar message [ 798.574504] Lustre: server umount lustre-MDT0000 complete [ 799.713600] LustreError: 24269:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 799.729084] LustreError: 24269:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 801.869308] LustreError: 24249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784186518 with bad export cookie 4761393657075950996 [ 801.876674] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 801.882855] LustreError: 24249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 811.643723] Lustre: server umount lustre-MDT0001 complete [ 826.212310] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 828.896985] Key type lgssc unregistered [ 829.328468] LNet: 28243:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 829.340345] LNetError: 28243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 829.365431] LNet: Removed LNI 192.168.203.142@tcp [ 830.363521] Key type .llcrypt unregistered [ 830.370155] Key type ._llcrypt unregistered [ 854.127465] Key type ._llcrypt registered [ 854.129112] Key type .llcrypt registered [ 854.247764] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 855.178452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 855.283603] alg: No test for adler32 (adler32-zlib) [ 856.610880] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 856.874196] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 858.592184] Key type lgssc registered [ 859.584657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 867.134889] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 867.151843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 868.575265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 872.772230] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 878.063133] LustreError: 29649:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 878.084709] LustreError: 29649:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 881.466097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 881.914290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 886.180218] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 893.369676] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 898.593430] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 912.537771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 912.948254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 913.964767] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 919.435438] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 927.270264] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 934.370239] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 934.376986] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 934.390647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 936.418253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 936.419762] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 936.449265] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 940.246920] Lustre: server umount lustre-OST0000 complete [ 946.658100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 946.685242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 946.695558] Lustre: Skipped 2 previous similar messages [ 950.311412] Lustre: server umount lustre-MDT0000 complete [ 951.778550] LustreError: 29645:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 951.804358] LustreError: 29645:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 954.151436] LustreError: 29629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784186671 with bad export cookie 15743014180190394183 [ 954.162489] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 954.166894] LustreError: 29629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 954.529578] Lustre: server umount lustre-MDT0001 complete [ 965.121223] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 967.776031] Key type lgssc unregistered [ 968.037672] LNet: 33527:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 968.045569] LNetError: 33527:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 968.069569] LNet: Removed LNI 192.168.203.142@tcp [ 968.938570] Key type .llcrypt unregistered [ 968.942361] Key type ._llcrypt unregistered [ 988.617707] Key type ._llcrypt registered [ 988.620959] Key type .llcrypt registered [ 988.719674] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 990.032207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 990.045486] alg: No test for adler32 (adler32-zlib) [ 991.137377] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 991.413247] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 993.112181] Key type lgssc registered [ 994.372498] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1001.537614] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1001.557416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1003.066403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1007.595550] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1012.199775] LustreError: 34922:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1012.215272] LustreError: 34922:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1016.343804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1016.650720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1020.761817] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1028.624292] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1034.751654] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1041.657559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1041.973248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1047.093612] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1047.913067] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1056.460430] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1067.490909] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1067.498275] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1067.518431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1067.526435] Lustre: Skipped 1 previous similar message [ 1071.839315] Lustre: server umount lustre-OST0000 complete [ 1077.730445] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1077.735264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1077.747654] Lustre: Skipped 2 previous similar messages [ 1081.489939] Lustre: server umount lustre-MDT0000 complete [ 1082.850271] LustreError: 34918:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1082.880671] LustreError: 34918:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1084.708415] LustreError: 34903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784186801 with bad export cookie 4193642186143254597 [ 1084.710526] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1084.720319] LustreError: 34903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1084.999981] Lustre: server umount lustre-MDT0001 complete [ 1097.522476] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 1100.256633] Key type lgssc unregistered [ 1100.605747] LNet: 38458:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1100.618614] LNetError: 38458:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1100.636973] LNet: Removed LNI 192.168.203.142@tcp [ 1101.842074] Key type .llcrypt unregistered [ 1101.846912] Key type ._llcrypt unregistered [ 1126.197153] Key type ._llcrypt registered [ 1126.200071] Key type .llcrypt registered [ 1126.411907] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1128.356695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1128.430968] alg: No test for adler32 (adler32-zlib) [ 1129.602394] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 1130.047846] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 1131.760580] Key type lgssc registered [ 1133.115939] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1141.692384] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1141.716943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1143.273263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1148.272828] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1153.508933] LustreError: 39872:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1153.530195] LustreError: 39872:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1157.392178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1157.888557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1162.439337] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1169.613484] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1174.893857] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1181.566628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1181.961156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1188.223772] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1189.064402] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1197.026048] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1203.682558] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1203.683320] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1203.691167] LustreError: Skipped 1 previous similar message [ 1203.693244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1203.719805] Lustre: Skipped 1 previous similar message [ 1208.805950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1208.811968] Lustre: Skipped 2 previous similar messages [ 1209.166844] Lustre: server umount lustre-OST0000 complete [ 1228.393132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1228.408698] Lustre: Skipped 1 previous similar message [ 1228.410489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1228.417096] Lustre: Skipped 2 previous similar messages [ 1233.569879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1233.576746] Lustre: Skipped 1 previous similar message [ 1234.745870] Lustre: server umount lustre-MDT0000 complete [ 1238.381483] LustreError: 39852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784186955 with bad export cookie 3452283934405055361 [ 1238.390813] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1238.396823] LustreError: 39852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1244.885157] Lustre: server umount lustre-MDT0001 complete [ 1256.374455] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 1259.572750] Key type lgssc unregistered [ 1259.974611] LNet: 43578:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1259.983705] LNetError: 43578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1261.030592] LNet: Removed LNI 192.168.203.142@tcp [ 1262.091101] Key type .llcrypt unregistered [ 1262.095637] Key type ._llcrypt unregistered [ 1285.631949] Key type ._llcrypt registered [ 1285.634802] Key type .llcrypt registered [ 1285.752719] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1286.902185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1286.923499] alg: No test for adler32 (adler32-zlib) [ 1287.954951] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 1288.182237] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 1289.896154] Key type lgssc registered [ 1290.890826] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1298.042087] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1298.071182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1299.456295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1303.334449] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1309.665898] LustreError: 44991:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1309.683404] LustreError: 44991:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1310.970681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1311.288203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1315.595519] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1324.177232] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1331.377624] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1338.156760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1338.528752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1344.039586] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1344.683855] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1350.930435] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1362.407234] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1362.415173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1362.416367] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1362.416893] Lustre: Skipped 2 previous similar messages [ 1362.450442] Lustre: Skipped 2 previous similar messages [ 1362.964379] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.42@tcp (stopping) [ 1366.320458] Lustre: server umount lustre-MDT0000 complete [ 1367.524112] LustreError: 47573:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1367.545040] LustreError: 47573:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1369.598185] LustreError: 44972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784187086 with bad export cookie 16309817972807726666 [ 1369.599966] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1369.607231] LustreError: 44972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1372.641159] LustreError: 44988:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1372.642105] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1372.672350] LustreError: 44988:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1372.694180] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1374.690223] LustreError: 47483:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1374.694139] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1374.711014] LustreError: 47483:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1374.727209] Lustre: Skipped 1 previous similar message [ 1376.070699] Lustre: server umount lustre-MDT0001 complete [ 1404.384165] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1404.602671] Lustre: server umount lustre-OST0000 complete [ 1420.338802] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 1422.862703] Key type lgssc unregistered [ 1423.200620] LNet: 48855:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1423.206817] LNetError: 48855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1424.230270] LNet: Removed LNI 192.168.203.142@tcp [ 1424.916601] Key type .llcrypt unregistered [ 1424.918958] Key type ._llcrypt unregistered [ 1446.755219] Key type ._llcrypt registered [ 1446.759602] Key type .llcrypt registered [ 1446.901271] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1447.862516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1447.880482] alg: No test for adler32 (adler32-zlib) [ 1448.900473] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 1449.162636] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 1450.856538] Key type lgssc registered [ 1451.767859] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1457.876720] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1457.904593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1473.440528] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1478.561043] LustreError: 50221:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.142@tcp: failed processing log, type 4: rc = -110 [ 1505.248220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1512.226784] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1519.850679] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1686.187655] Lustre: server umount lustre-OST0000 complete [ 1702.664913] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 1705.453674] Key type lgssc unregistered [ 1705.800829] LNet: 51900:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1705.810547] LNetError: 51900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1705.839891] LNet: Removed LNI 192.168.203.142@tcp [ 1706.736873] Key type .llcrypt unregistered [ 1706.741584] Key type ._llcrypt unregistered [ 1731.330068] Key type ._llcrypt registered [ 1731.333434] Key type .llcrypt registered [ 1731.436344] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1732.788612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1732.854607] alg: No test for adler32 (adler32-zlib) [ 1733.878419] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 1734.105415] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 1735.768139] Key type lgssc registered [ 1736.831282] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1744.229951] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1744.246148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1745.648592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1749.336466] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1755.112274] LustreError: 53317:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1755.127099] LustreError: 53317:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1757.480711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1757.893554] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1762.084584] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1769.869509] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1776.329507] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1783.578653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1783.963815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1789.560805] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1790.143420] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1796.562432] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1800.675446] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1800.711493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1802.801949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1807.854157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1807.865476] Lustre: Skipped 2 previous similar messages [ 1812.960954] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1812.963418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1812.996866] Lustre: Skipped 1 previous similar message [ 1813.405632] Lustre: server umount lustre-OST0000 complete [ 1818.095649] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1818.102368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1818.106923] Lustre: Skipped 1 previous similar message [ 1818.125692] Lustre: Skipped 1 previous similar message [ 1823.556886] Lustre: server umount lustre-MDT0000 complete [ 1827.084092] LustreError: 53298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784187544 with bad export cookie 2705969683040172856 [ 1827.090673] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1827.093930] LustreError: 53298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1827.501239] Lustre: server umount lustre-MDT0001 complete [ 1838.572952] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 1841.200428] Key type lgssc unregistered [ 1841.478961] LNet: 56846:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1841.487843] LNetError: 56846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1841.505703] LNet: Removed LNI 192.168.203.142@tcp [ 1842.324602] Key type .llcrypt unregistered [ 1842.326166] Key type ._llcrypt unregistered [ 1863.737601] Key type ._llcrypt registered [ 1863.740805] Key type .llcrypt registered [ 1863.835611] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1864.833820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1864.874658] alg: No test for adler32 (adler32-zlib) [ 1865.974719] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 1866.261329] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 1867.937115] Key type lgssc registered [ 1868.924126] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1875.307082] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1875.336641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1890.913680] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1896.032328] LustreError: 58233:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.142@tcp: failed processing log, type 4: rc = -110 [ 1922.720601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1928.142418] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1934.869635] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1940.954777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1941.493512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1941.547019] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1945.449207] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1946.594516] LustreError: 58258:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1946.611017] LustreError: 58258:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1951.717464] LustreError: 58259:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1952.878767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1953.095651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1956.392181] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1963.957406] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1971.210303] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1977.316193] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1977.324271] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1977.331359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1977.863914] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1977.880165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1980.393083] Lustre: server umount lustre-OST0000 complete [ 2004.451306] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2004.467067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2004.485868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2007.523835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2007.524597] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2010.396592] Lustre: server umount lustre-MDT0000 complete [ 2012.646566] LustreError: 59272:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2012.672921] LustreError: 59272:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2013.421160] LustreError: 58240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784187730 with bad export cookie 8286556485015710346 [ 2013.432134] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2013.437138] LustreError: 58240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2019.732582] Lustre: server umount lustre-MDT0001 complete [ 2031.120406] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 2033.427970] Key type lgssc unregistered [ 2033.704674] LNet: 61979:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2033.710593] LNetError: 61979:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2033.733819] LNet: Removed LNI 192.168.203.142@tcp [ 2034.458401] Key type .llcrypt unregistered [ 2034.461082] Key type ._llcrypt unregistered [ 2054.462804] Key type ._llcrypt registered [ 2054.466735] Key type .llcrypt registered [ 2054.571123] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 2055.845046] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2055.870202] alg: No test for adler32 (adler32-zlib) [ 2057.072896] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 2057.396735] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 2059.116262] Key type lgssc registered [ 2060.407784] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2067.478267] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2067.495507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2068.868383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2072.683804] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2078.199473] LustreError: 63375:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2078.223362] LustreError: 63375:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2080.674463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2080.948361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2085.300427] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2092.330402] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2097.565230] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2104.383806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2104.899914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2112.027196] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2113.144565] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2119.597823] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2137.063255] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2137.077331] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2137.102720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2139.109058] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2139.116948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2139.116961] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2141.087180] Lustre: server umount lustre-OST0000 complete [ 2147.812289] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2147.816604] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2147.840180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2147.844737] Lustre: Skipped 1 previous similar message [ 2150.863880] Lustre: server umount lustre-MDT0000 complete [ 2154.468303] LustreError: 65557:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2154.494704] LustreError: 65557:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2154.635880] LustreError: 63354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784187871 with bad export cookie 9184198317821147684 [ 2154.638420] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2154.648965] LustreError: 63354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2155.042974] Lustre: server umount lustre-MDT0001 complete [ 2165.630219] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 2168.055821] Key type lgssc unregistered [ 2168.401031] LNet: 66933:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2168.411964] LNetError: 66933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2168.426055] LNet: Removed LNI 192.168.203.142@tcp [ 2169.362334] Key type .llcrypt unregistered [ 2169.365151] Key type ._llcrypt unregistered [ 2194.039970] Key type ._llcrypt registered [ 2194.042840] Key type .llcrypt registered [ 2194.144495] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 2194.951841] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2195.142369] alg: No test for adler32 (adler32-zlib) [ 2196.222961] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 2196.508770] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 2198.184186] Key type lgssc registered [ 2199.035117] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2207.417138] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 03:45:22 (1784187922) [ 2213.335342] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2213.357601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2214.763803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2218.711584] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2224.112277] LustreError: 68638:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2224.127668] LustreError: 68638:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2226.165054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2226.485113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2231.260332] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2240.109726] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2247.234670] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2254.393337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2254.753810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2259.911209] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2261.303509] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2268.295647] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2277.751423] Lustre: Failing over lustre-MDT0000 [ 2277.862481] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2277.875553] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2277.900573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2278.033928] Lustre: server umount lustre-MDT0000 complete [ 2280.986927] LustreError: 70479:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2281.453560] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2281.482897] Lustre: Skipped 1 previous similar message [ 2286.074218] LustreError: 68634:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2286.090810] LustreError: 68634:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2287.112812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2287.306398] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2287.492177] Lustre: *** cfs_fail_loc=135, val=0*** [ 2287.627438] LustreError: 71725:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2287.639038] LustreError: 71725:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.142@tcp: cfg command failed: rc = -2 [ 2287.646405] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2287.646405] [ 2287.652894] LustreError: MGC192.168.203.142@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2287.673349] LustreError: 71714:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2287.682366] LustreError: 71714:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2287.690332] LustreError: 71714:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2287.694829] LustreError: 71714:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2287.723395] Lustre: server umount lustre-MDT0000 complete [ 2287.726194] LustreError: 71714:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2291.197339] LustreError: 68632:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2291.221702] LustreError: 68632:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2294.516976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2294.670888] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2295.006266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2295.074472] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2296.318629] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2299.879950] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2300.396533] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2300.432104] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2300.477126] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2310.632105] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2310.647064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2310.651796] Lustre: Skipped 1 previous similar message [ 2310.666528] Lustre: Skipped 1 previous similar message [ 2314.513349] Lustre: server umount lustre-OST0000 complete [ 2320.865467] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2320.866827] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2320.874587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2320.891828] Lustre: Skipped 1 previous similar message [ 2320.903602] Lustre: Skipped 1 previous similar message [ 2324.364408] Lustre: server umount lustre-MDT0000 complete [ 2325.987712] LustreError: 68637:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2326.001378] LustreError: 68637:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2327.815786] LustreError: 70481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784188044 with bad export cookie 7142666876886649456 [ 2327.816413] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2327.827590] LustreError: 70481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2328.101841] Lustre: server umount lustre-MDT0001 complete [ 2337.597312] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 2339.953418] Key type lgssc unregistered [ 2340.156314] LNet: 73502:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2340.163199] LNetError: 73502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2340.183181] LNet: Removed LNI 192.168.203.142@tcp [ 2341.037182] Key type .llcrypt unregistered [ 2341.041325] Key type ._llcrypt unregistered [ 2364.232630] Key type ._llcrypt registered [ 2364.240837] Key type .llcrypt registered [ 2364.334840] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 2365.106445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2365.124505] alg: No test for adler32 (adler32-zlib) [ 2366.182515] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 2366.460028] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 2368.184168] Key type lgssc registered [ 2369.160714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2376.686702] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2376.711270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2378.143436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2382.862508] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2388.452140] LustreError: 74918:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2388.471121] LustreError: 74918:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2391.443888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2391.835118] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2396.465329] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2404.516611] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2411.197368] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2418.063993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2418.367696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2423.551432] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2423.675412] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2431.410179] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2441.593985] Lustre: Failing over lustre-MDT0000 [ 2442.211266] Lustre: server umount lustre-MDT0000 complete [ 2442.214667] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2442.218888] LustreError: 74915:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2442.273328] Lustre: Skipped 1 previous similar message [ 2444.325873] LustreError: 75212:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2444.370064] LustreError: 75212:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2447.331066] LustreError: 75647:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2447.350364] LustreError: 75647:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2450.920285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2451.085982] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2451.223070] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2451.345847] LustreError: 78004:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2451.353550] LustreError: 78004:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.142@tcp: cfg command failed: rc = -9 [ 2451.370840] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2451.370840] [ 2451.378641] LustreError: MGC192.168.203.142@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2451.392851] LustreError: 77992:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2451.397912] LustreError: 77992:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2451.407724] LustreError: 77992:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2451.416475] LustreError: 77992:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2451.433218] Lustre: server umount lustre-MDT0000 complete [ 2451.436921] LustreError: 77992:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2452.452868] LustreError: 74918:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2452.483565] LustreError: 74918:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2458.095089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2458.279820] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2458.564074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2458.628691] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2459.651721] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2462.720088] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2463.749707] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2463.796039] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2463.830148] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2472.940904] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2472.959627] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2472.977408] Lustre: Skipped 1 previous similar message [ 2472.984730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2473.969396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2473.972883] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2476.253573] Lustre: server umount lustre-OST0000 complete [ 2484.201319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2484.212046] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2484.216825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2484.216833] Lustre: Skipped 1 previous similar message [ 2486.668365] Lustre: server umount lustre-MDT0000 complete [ 2489.314877] LustreError: 74914:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2489.338581] LustreError: 74914:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2490.877277] LustreError: 74900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784188207 with bad export cookie 1269120272909680303 [ 2490.895508] LustreError: 74900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2490.895898] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2491.240928] Lustre: server umount lustre-MDT0001 complete [ 2502.426932] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 2505.046502] Key type lgssc unregistered [ 2505.337479] LNet: 79787:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2505.346063] LNetError: 79787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2505.367916] LNet: Removed LNI 192.168.203.142@tcp [ 2506.063850] Key type .llcrypt unregistered [ 2506.065828] Key type ._llcrypt unregistered [ 2519.296861] Key type ._llcrypt registered [ 2519.300807] Key type .llcrypt registered [ 2519.438918] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 2519.722359] Key type .llcrypt unregistered [ 2519.724167] Key type ._llcrypt unregistered [ 2541.785191] Key type ._llcrypt registered [ 2541.789164] Key type .llcrypt registered [ 2541.905911] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 2543.068208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2543.102829] alg: No test for adler32 (adler32-zlib) [ 2544.133574] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 2544.383830] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 2546.080122] Key type lgssc registered [ 2547.022715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2554.557768] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2554.598230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2555.938432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2560.650737] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2565.096379] LustreError: 81690:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2565.112900] LustreError: 81690:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2569.339382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2569.769100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2574.587908] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2583.654323] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2591.254444] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2601.599519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2602.027342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2610.204284] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2617.529295] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2625.598718] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2639.844668] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2639.855168] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2639.867222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2640.867384] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2640.875699] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2640.877982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2642.153760] Lustre: server umount lustre-OST0000 complete [ 2645.987668] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2645.992853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2646.011992] Lustre: Skipped 1 previous similar message [ 2646.024918] Lustre: Skipped 2 previous similar messages [ 2651.105356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2651.112988] Lustre: Skipped 1 previous similar message [ 2656.229977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2656.239027] Lustre: Skipped 1 previous similar message [ 2660.320679] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2660.544225] Lustre: server umount lustre-MDT0000 complete [ 2661.347496] LustreError: 81690:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2661.372128] LustreError: 81690:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2664.031496] LustreError: 81671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784188380 with bad export cookie 17057347147247639284 [ 2664.032463] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2664.038430] LustreError: 81671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2664.551972] Lustre: server umount lustre-MDT0001 complete [ 2675.073568] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 2677.474707] Key type lgssc unregistered [ 2677.780789] LNet: 85245:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2677.791235] LNetError: 85245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2677.813865] LNet: Removed LNI 192.168.203.142@tcp [ 2678.641678] Key type .llcrypt unregistered [ 2678.646722] Key type ._llcrypt unregistered [ 2704.578420] Key type ._llcrypt registered [ 2704.586272] Key type .llcrypt registered [ 2704.769849] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 2705.617149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2705.658792] alg: No test for adler32 (adler32-zlib) [ 2706.879465] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 2707.117762] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 2708.808411] Key type lgssc registered [ 2710.442402] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2719.365947] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2719.382315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2720.964470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2725.839243] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2730.469491] LustreError: 86659:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2730.490471] LustreError: 86659:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2735.355799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2735.588227] LustreError: 86658:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2735.814058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2739.817741] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2747.272482] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2753.422742] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2760.257476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2760.644870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2765.759925] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2766.792734] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2773.839486] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2781.154773] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2781.166582] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2781.168834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2781.186341] Lustre: Skipped 1 previous similar message [ 2781.196820] Lustre: Skipped 1 previous similar message [ 2781.794728] Lustre: server umount lustre-OST0000 complete [ 2786.275922] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2786.278587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2786.287520] Lustre: Skipped 1 previous similar message [ 2791.137230] Lustre: server umount lustre-MDT0000 complete [ 2791.393737] LustreError: 86654:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2791.422382] LustreError: 86654:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2794.273092] LustreError: 88502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784188511 with bad export cookie 3152952058199723559 [ 2794.279249] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2794.280583] LustreError: 88502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2794.496269] Lustre: server umount lustre-MDT0001 complete [ 2803.543578] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 2806.343050] Key type lgssc unregistered [ 2806.624859] LNet: 90189:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2806.646329] LNetError: 90189:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2806.666636] LNet: Removed LNI 192.168.203.142@tcp [ 2807.526745] Key type .llcrypt unregistered [ 2807.529261] Key type ._llcrypt unregistered [ 2828.968214] Key type ._llcrypt registered [ 2828.976891] Key type .llcrypt registered [ 2829.154626] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 2830.416970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2830.493805] alg: No test for adler32 (adler32-zlib) [ 2831.581986] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 2831.894886] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 2833.608939] Key type lgssc registered [ 2835.154067] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2843.054388] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2843.067868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2844.440856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2848.250712] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2854.882208] LustreError: 91603:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2854.911237] LustreError: 91603:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2856.260474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2856.627517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2861.364611] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2868.239978] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2873.768844] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2880.953082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2881.425187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2888.697756] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2889.630720] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2897.136281] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2908.131894] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2908.146869] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2908.168529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2910.178831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2910.193324] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2910.209047] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2914.277117] Lustre: server umount lustre-OST0000 complete [ 2918.378420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2918.391771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2918.416375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2918.429109] Lustre: Skipped 1 previous similar message [ 2920.423400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2920.423849] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2925.543476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2925.548617] Lustre: Skipped 2 previous similar messages [ 2932.193188] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2932.352514] Lustre: server umount lustre-MDT0000 complete [ 2935.790899] LustreError: 93091:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2935.803533] LustreError: 93091:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2936.131732] LustreError: 91583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784188653 with bad export cookie 14333860503036493019 [ 2936.135730] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2936.143218] LustreError: 91583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2936.449949] Lustre: server umount lustre-MDT0001 complete [ 2947.346351] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 2950.189319] Key type lgssc unregistered [ 2950.569082] LNet: 95137:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2950.577231] LNetError: 95137:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2951.661609] LNet: Removed LNI 192.168.203.142@tcp [ 2952.695923] Key type .llcrypt unregistered [ 2952.697661] Key type ._llcrypt unregistered [ 2977.062622] Key type ._llcrypt registered [ 2977.065714] Key type .llcrypt registered [ 2977.202506] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 2978.517187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2978.531310] alg: No test for adler32 (adler32-zlib) [ 2979.590664] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 2979.886444] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 2981.560202] Key type lgssc registered [ 2982.783736] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2990.579762] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2990.603521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3006.176525] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3011.297123] LustreError: 96523:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.142@tcp: failed processing log, type 4: rc = -110 [ 3037.984311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3044.690864] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3051.560872] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3057.077873] Lustre: server umount lustre-OST0000 complete [ 3066.652565] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 03:59:41 (1784188781) [ 3073.334733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3073.713707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3077.863935] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3079.138474] LustreError: 98163:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3079.155926] LustreError: 98163:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3083.246651] LustreError: 98162:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3086.872289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3087.344607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3092.488774] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3101.017742] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3107.518316] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3115.160371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3115.633533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3118.718984] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3122.389828] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3131.962606] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3143.657906] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3143.667058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3143.674331] Lustre: Skipped 1 previous similar message [ 3143.678502] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3143.705698] Lustre: Skipped 1 previous similar message [ 3147.184425] Lustre: server umount lustre-OST0000 complete [ 3153.889972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3153.893270] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3153.907156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3153.931198] Lustre: Skipped 1 previous similar message [ 3153.952505] Lustre: Skipped 1 previous similar message [ 3156.581439] Lustre: server umount lustre-MDT0000 complete [ 3159.009159] LustreError: 98162:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3159.044153] LustreError: 98162:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3159.921869] LustreError: 100004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784188876 with bad export cookie 17626780201499545159 [ 3159.930988] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3159.934597] LustreError: 100004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3160.190469] Lustre: server umount lustre-MDT0001 complete [ 3172.009479] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 3174.705923] Key type lgssc unregistered [ 3175.074649] LNet: 101869:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3175.082867] LNetError: 101869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3175.119382] LNet: Removed LNI 192.168.203.142@tcp [ 3175.892168] Key type .llcrypt unregistered [ 3175.894452] Key type ._llcrypt unregistered [ 3213.871554] Key type ._llcrypt registered [ 3213.874169] Key type .llcrypt registered [ 3213.983387] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 3214.845976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3215.012312] alg: No test for adler32 (adler32-zlib) [ 3216.071264] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 3216.386267] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 3218.064229] Key type lgssc registered [ 3219.174675] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3226.974156] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3226.991960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3228.359922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3232.671427] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3238.882829] LustreError: 104033:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3238.912519] LustreError: 104033:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3240.069447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3240.424438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3245.057149] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3251.855296] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3257.155423] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3264.138859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3264.521199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3270.414813] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3273.713558] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3277.146988] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3285.985245] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3285.989096] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3286.004643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3289.061060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3289.067309] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3289.078927] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3290.820540] Lustre: server umount lustre-OST0000 complete [ 3296.746533] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3296.755696] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3296.777556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3296.780202] Lustre: Skipped 1 previous similar message [ 3299.297588] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3299.300120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3299.327542] Lustre: Skipped 1 previous similar message [ 3300.972466] Lustre: server umount lustre-MDT0000 complete [ 3304.418815] LustreError: 104033:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3304.429803] LustreError: 104033:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3304.734773] LustreError: 104016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784189021 with bad export cookie 10604187657366177579 [ 3304.741939] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3304.742836] LustreError: 104016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3305.121329] Lustre: server umount lustre-MDT0001 complete [ 3314.478300] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 3316.749217] Key type lgssc unregistered [ 3317.004363] LNet: 107562:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3317.015089] LNetError: 107562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3318.063576] LNet: Removed LNI 192.168.203.142@tcp [ 3318.805161] Key type .llcrypt unregistered [ 3318.809422] Key type ._llcrypt unregistered [ 3335.165180] Key type ._llcrypt registered [ 3335.169903] Key type .llcrypt registered [ 3335.286516] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 3336.699415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3336.724273] alg: No test for adler32 (adler32-zlib) [ 3337.775434] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 3338.023537] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 3339.704170] Key type lgssc registered [ 3340.742992] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3347.008443] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3347.024814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3362.533343] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3367.648307] LustreError: 108799:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.142@tcp: failed processing log, type 4: rc = -110 [ 3394.336303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3400.031038] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3406.545897] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3412.367383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3412.658305] LustreError: MGC192.168.203.142@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3412.668074] LustreError: 109822:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3412.677573] LustreError: 109822:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3412.685681] LustreError: 109822:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3412.749561] Lustre: server umount lustre-MDT0000 complete [ 3412.754746] LustreError: 109822:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3430.543191] Lustre: server umount lustre-OST0000 complete [ 3441.178166] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 3449.288566] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 3485.179232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3485.396147] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3485.426639] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3485.518582] Lustre: lustre-MDT0000: new disk, initializing [ 3485.595072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3485.627101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3490.383456] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3500.593668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3500.727286] Lustre: 112617:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3500.794893] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3500.799162] Lustre: Skipped 1 previous similar message [ 3500.923314] Lustre: lustre-MDT0001: new disk, initializing [ 3501.046247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3501.097681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3501.106544] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3505.028653] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3513.642750] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3518.344493] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3523.996195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3524.239755] Lustre: lustre-OST0000: new disk, initializing [ 3524.243304] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3524.253824] Lustre: 114583:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3524.343623] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3530.318721] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3532.889129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3532.904568] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3533.039923] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3539.428854] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3543.588514] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3543.759325] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3548.219836] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3548.433072] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3553.762084] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3553.764868] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3553.767349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3553.780377] Lustre: Skipped 1 previous similar message [ 3553.796081] Lustre: Skipped 1 previous similar message [ 3556.305116] Lustre: server umount lustre-OST0000 complete [ 3562.466113] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3562.469943] LustreError: Skipped 1 previous similar message [ 3562.477360] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3562.490630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3564.006049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3564.008335] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3565.992660] Lustre: server umount lustre-MDT0000 complete [ 3569.621202] LustreError: 112609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784189286 with bad export cookie 16865294942518117032 [ 3569.630451] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3569.631384] LustreError: 112609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3570.011981] Lustre: server umount lustre-MDT0001 complete [ 3578.140684] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 04:08:12 (1784189292) [ 3581.590285] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3597.610935] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 3606.842358] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 3641.778936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3642.017041] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3642.040768] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3642.099448] Lustre: lustre-MDT0000: new disk, initializing [ 3642.154662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3642.165258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3646.677663] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3657.450960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3657.600593] Lustre: 119488:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3657.670258] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3657.677493] Lustre: Skipped 1 previous similar message [ 3657.769709] Lustre: lustre-MDT0001: new disk, initializing [ 3657.912034] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3657.969727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3657.982276] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3662.227282] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3671.922691] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3677.761793] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3683.923405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3684.180768] Lustre: lustre-OST0000: new disk, initializing [ 3684.187052] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3684.193165] Lustre: 121459:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3684.262130] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3689.524047] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3692.092350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3692.099060] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3692.207023] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3698.403824] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3702.389670] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3702.601437] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3706.658464] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3706.913444] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3713.001505] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3713.017895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3713.026354] Lustre: Skipped 1 previous similar message [ 3713.045400] Lustre: Skipped 2 previous similar messages [ 3718.115523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3723.232162] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3723.481662] Lustre: server umount lustre-OST0000 complete [ 3728.354727] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3728.370282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3728.383452] Lustre: Skipped 4 previous similar messages [ 3741.152223] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3741.388101] Lustre: server umount lustre-MDT0000 complete [ 3743.713476] LustreError: 119502:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3743.732054] LustreError: 119502:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3745.090439] LustreError: 119482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784189462 with bad export cookie 16865294942518117914 [ 3745.092125] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3745.104296] LustreError: 119482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3745.572725] Lustre: server umount lustre-MDT0001 complete [ 3754.769728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3755.356733] LustreError: 123677:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3755.458749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3760.613980] LustreError: 123678:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3760.741916] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3765.736121] LustreError: 123677:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3768.581281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3772.935586] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3780.316814] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3786.790932] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3794.612748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3801.670902] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3809.592193] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3819.528432] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3824.613359] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3824.635284] Lustre: Skipped 1 previous similar message [ 3824.639187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3824.658845] Lustre: Skipped 5 previous similar messages [ 3829.603853] Lustre: server umount lustre-OST0000 complete [ 3839.615243] Lustre: server umount lustre-MDT0000 complete [ 3839.970134] LustreError: 123672:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3839.985542] LustreError: 123672:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3843.480336] LustreError: 126558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784189560 with bad export cookie 16865294942518118796 [ 3843.481502] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3843.487162] LustreError: 126558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3843.951849] Lustre: server umount lustre-MDT0001 complete [ 3855.168995] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 3857.901212] Key type lgssc unregistered [ 3858.117676] LNet: 127310:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3858.130871] LNetError: 127310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3858.158989] LNet: Removed LNI 192.168.203.142@tcp [ 3858.986602] Key type .llcrypt unregistered [ 3858.988651] Key type ._llcrypt unregistered [ 3880.190373] Key type ._llcrypt registered [ 3880.193681] Key type .llcrypt registered [ 3880.311503] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 3896.145482] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 3897.435814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3897.481396] alg: No test for adler32 (adler32-zlib) [ 3898.612490] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 3898.941634] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 3900.664162] Key type lgssc registered [ 3901.990598] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3938.879461] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3938.924470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3940.324383] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3940.367694] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3940.491951] Lustre: lustre-MDT0000: new disk, initializing [ 3940.621289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3940.643147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3946.022218] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3957.460441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3957.621482] Lustre: 130465:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3957.689586] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3957.699328] Lustre: Skipped 1 previous similar message [ 3957.791337] Lustre: lustre-MDT0001: new disk, initializing [ 3957.913117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3957.987283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3958.011231] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3963.459494] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3973.110966] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3979.015357] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3985.635404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3985.850500] Lustre: lustre-OST0000: new disk, initializing [ 3985.855575] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3985.860979] Lustre: 132433:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3985.953527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3991.615228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3991.637566] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3991.794113] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3992.498395] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4002.243260] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4008.082445] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4008.365882] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4013.233962] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4013.487431] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4017.634232] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4017.652239] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4017.668597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4021.723898] Lustre: server umount lustre-OST0000 complete [ 4027.874666] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4027.878968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4027.890607] Lustre: Skipped 2 previous similar messages [ 4027.914165] Lustre: Skipped 2 previous similar messages [ 4031.626363] Lustre: server umount lustre-MDT0000 complete [ 4035.583595] LustreError: 130456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784189752 with bad export cookie 565974080313967847 [ 4035.591418] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4035.602781] LustreError: 130456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4035.981312] Lustre: server umount lustre-MDT0001 complete [ 4043.711987] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 04:15:58 (1784189758) [ 4049.650730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4050.139277] LustreError: 134847:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4050.164175] LustreError: 134847:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4050.211697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4054.204729] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4055.523506] LustreError: 134848:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4060.651709] LustreError: 134847:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4062.825239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4063.091715] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4067.370818] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4077.516708] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4084.262667] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4088.812423] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4088.834417] LustreError: Skipped 1 previous similar message [ 4088.846034] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4088.883513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4093.326843] Lustre: server umount lustre-MDT0000 complete [ 4094.945482] LustreError: 135859:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4094.961274] LustreError: 135859:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4097.515858] LustreError: 134829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784189814 with bad export cookie 565974080313968729 [ 4097.523435] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4097.532337] LustreError: 134829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4098.046881] Lustre: server umount lustre-MDT0001 complete [ 4107.634848] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 04:17:02 (1784189822) [ 4115.361149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4131.040631] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4136.161397] LustreError: 137263:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.142@tcp: failed processing log, type 4: rc = -110 [ 4161.888600] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4169.869899] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4180.500091] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4182.654668] Lustre: server umount lustre-OST0000 complete [ 4192.644385] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 04:18:27 (1784189907) [ 4200.435694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4200.813681] LustreError: 138710:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4200.884868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4205.129822] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4210.154232] LustreError: 138711:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4210.170128] LustreError: 138711:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4213.432435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4217.992675] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4225.476840] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4230.998833] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4237.640393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4238.086620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4238.089684] Lustre: Skipped 1 previous similar message [ 4244.325216] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4252.004171] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4259.810487] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4259.820332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4259.844128] Lustre: Skipped 1 previous similar message [ 4259.853731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4259.861260] Lustre: Skipped 2 previous similar messages [ 4265.944753] Lustre: server umount lustre-MDT0000 complete [ 4265.960203] LustreError: 139735:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4265.978083] LustreError: 139735:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4270.138466] LustreError: 141292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784189987 with bad export cookie 565974080313969387 [ 4270.143829] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4270.156216] LustreError: 141292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4270.525290] Lustre: server umount lustre-MDT0001 complete [ 4284.918452] Lustre: server umount lustre-OST0000 complete [ 4293.681611] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 04:20:08 (1784190008) [ 4300.401408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4300.902258] LustreError: 142365:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4305.442427] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4314.033651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4314.525433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4314.535401] Lustre: Skipped 1 previous similar message [ 4319.380784] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4327.975692] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4334.147302] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4341.112273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4345.521823] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4347.579852] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4356.050600] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4360.822097] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4361.048307] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4366.122334] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4366.432406] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4370.403569] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4370.414771] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4370.422725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4370.423212] Lustre: Skipped 3 previous similar messages [ 4370.428484] Lustre: Skipped 3 previous similar messages [ 4374.839403] Lustre: server umount lustre-OST0000 complete [ 4380.641809] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4380.669417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4380.672879] Lustre: Skipped 1 previous similar message [ 4380.691493] Lustre: Skipped 2 previous similar messages [ 4385.546829] Lustre: server umount lustre-MDT0000 complete [ 4385.763570] LustreError: 142366:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4385.794542] LustreError: 142366:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4389.550881] LustreError: 142347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784190106 with bad export cookie 565974080313970465 [ 4389.561505] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4389.570263] LustreError: 142347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4390.119396] Lustre: server umount lustre-MDT0001 complete [ 4398.737884] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 04:21:53 (1784190113) [ 4405.858984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4421.600960] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4426.789264] LustreError: 146523:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.142@tcp: failed processing log, type 4: rc = -110 [ 4452.512628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4452.527128] Lustre: Skipped 1 previous similar message [ 4458.329258] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4465.645728] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4472.079046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4472.633129] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4476.633995] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4484.987901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4490.203609] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4499.027185] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4504.916782] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4509.483476] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4509.730038] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4514.281696] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4514.482897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4518.382324] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4518.393941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4518.405466] Lustre: Skipped 1 previous similar message [ 4518.421916] Lustre: Skipped 2 previous similar messages [ 4522.604548] Lustre: server umount lustre-MDT0000 complete [ 4523.495345] LustreError: 147558:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4523.510503] LustreError: 147558:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4526.149907] LustreError: 146529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784190243 with bad export cookie 565974080313971487 [ 4526.162215] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4526.172320] LustreError: 146529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4526.658868] Lustre: server umount lustre-MDT0001 complete [ 4541.064946] Lustre: server umount lustre-OST0000 complete [ 4548.716290] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 04:24:23 (1784190263) [ 4554.969388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4570.593485] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4575.712691] LustreError: 150719:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.142@tcp: failed processing log, type 4: rc = -110 [ 4608.148550] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4614.726912] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4620.554887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4621.053902] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4625.692146] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4633.993691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4639.314489] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4647.770840] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4654.124472] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4660.760890] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4660.922459] Lustre: lustre-OST0001: new disk, initializing [ 4660.927901] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4660.935898] Lustre: 153593:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4666.676653] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4671.063761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4671.072113] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4671.229042] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4676.522467] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4682.181253] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4682.506876] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4687.136925] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4687.379600] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4690.913676] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4690.927757] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4690.942928] Lustre: Skipped 1 previous similar message [ 4690.951078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4695.463509] Lustre: server umount lustre-OST0000 complete [ 4702.178185] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4702.191521] LustreError: Skipped 1 previous similar message [ 4705.267993] Lustre: server umount lustre-OST0001 complete [ 4723.170580] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4723.344831] Lustre: server umount lustre-MDT0000 complete [ 4726.833275] LustreError: 150726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784190443 with bad export cookie 565974080313972439 [ 4726.837196] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4726.849214] LustreError: 150726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4727.209725] Lustre: server umount lustre-MDT0001 complete [ 4752.785307] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 04:27:47 (1784190467) [ 4754.572417] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4756.637434] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4758.420544] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 04:27:53 (1784190473) [ 4764.939093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4765.265502] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4765.329195] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4765.562723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4765.566574] Lustre: Skipped 6 previous similar messages [ 4769.940315] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4778.307937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4778.434045] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4778.474732] Lustre: 157156:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4784.604320] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4792.900847] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4798.470692] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4805.696172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4806.009316] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4813.026297] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4814.502976] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4820.908773] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4826.939094] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4827.139799] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4832.131746] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4832.382529] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4834.789133] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4834.800438] LustreError: Skipped 1 previous similar message [ 4834.818076] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4834.841552] Lustre: Skipped 5 previous similar messages [ 4834.859231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4834.871303] Lustre: Skipped 11 previous similar messages [ 4840.463528] Lustre: server umount lustre-OST0000 complete [ 4863.232566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4872.181368] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4875.108926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4875.133350] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4881.111355] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4885.442402] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4885.468356] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4887.787155] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4888.096977] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4893.160233] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4893.402754] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4899.680958] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff912585c01000.ost_server_uuid 50 [ 4901.140610] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff912585c01000.ost_server_uuid in FULL state after 0 sec [ 4901.406288] LustreError: 159444:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.42@tcp) returned error from blocking AST (req@00000000c6e4e6bd x1870858410001536 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000008f40b5b4/0x7dabe6d69316031 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.42@tcp remote: 0x4210cdcfde923962 expref: 5 pid: 159444 timeout: 5001 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4901.472372] LustreError: lustre-MDT0000: A client on nid 192.168.203.42@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4901.489695] LustreError: 157151:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.42@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000008f40b5b4/0x7dabe6d69316031 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.42@tcp remote: 0x4210cdcfde923962 expref: 6 pid: 159444 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4906.990727] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4907.000155] LustreError: Skipped 1 previous similar message [ 4912.854139] Lustre: server umount lustre-OST0000 complete [ 4923.737987] Lustre: server umount lustre-MDT0000 complete [ 4926.437599] LustreError: 159444:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4926.465932] LustreError: 159444:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 4928.421237] LustreError: 157147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784190645 with bad export cookie 565974080313973615 [ 4928.432697] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4928.438479] LustreError: 157147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4934.928806] Lustre: server umount lustre-MDT0001 complete [ 4948.480874] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 4951.828763] Key type lgssc unregistered [ 4952.144951] LNet: 163267:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4952.157645] LNetError: 163267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4952.180151] LNet: Removed LNI 192.168.203.142@tcp [ 4953.095237] Key type .llcrypt unregistered [ 4953.100237] Key type ._llcrypt unregistered [ 4977.838980] Key type ._llcrypt registered [ 4977.841771] Key type .llcrypt registered [ 4977.928881] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 4978.849608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4978.984350] alg: No test for adler32 (adler32-zlib) [ 4980.012507] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 4980.253401] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 4981.880178] Key type lgssc registered [ 4982.788884] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4991.239700] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4991.257210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4992.961207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4998.427156] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5002.219742] LustreError: 164680:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5002.237756] LustreError: 164680:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5007.329169] LustreError: 164681:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5008.774040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5009.344754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5014.358164] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5021.900557] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5027.912927] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5034.653793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5034.973449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5038.076232] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5041.116598] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5048.855306] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5054.310531] Lustre: Failing over lustre-MDT0000 [ 5054.701459] Lustre: server umount lustre-MDT0000 complete [ 5055.465365] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5055.474691] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5055.489252] LustreError: 164681:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5058.529869] LustreError: 165408:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5058.533226] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5058.550121] LustreError: 165408:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5063.653936] LustreError: 164675:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5063.671687] LustreError: 164675:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 5066.758703] LustreError: 164661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.42@tcp arrived at 1784190783 with bad export cookie 2349095573911993411 [ 5066.781806] LustreError: 164661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5073.891915] LustreError: 166519:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5073.902296] LustreError: 166519:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5074.857926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5074.912813] Lustre: 163994:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784190775/real 1784190775] req@0000000023442a1e x1870859544324352/t0(0) o400->MGC192.168.203.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1784190791 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5074.970467] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5084.131301] LustreError: 163990:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e369c35d x1870859544329600/t0(0) o250->MGC192.168.203.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5084.502426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5084.583184] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5089.770842] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5089.782094] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5089.893140] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5092.271971] Lustre: lustre-MDT0000: Denying connection for new client fe5785dc-9f95-44c8-a4dc-4c3ab727d955 (at 192.168.203.42@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 5097.485553] Lustre: lustre-MDT0000: Denying connection for new client fe5785dc-9f95-44c8-a4dc-4c3ab727d955 (at 192.168.203.42@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:02 [ 5103.021985] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5103.029752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5103.039941] Lustre: Skipped 2 previous similar messages [ 5103.050839] Lustre: Skipped 1 previous similar message [ 5106.860363] Lustre: server umount lustre-OST0000 complete [ 5110.219132] LustreError: 168418:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5110.231384] Lustre: 167767:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5110.236212] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5110.257512] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5110.270362] LustreError: 167767:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000002f0d1b5 x1870859544344704/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5110.302164] LustreError: 167767:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5110.314054] LustreError: 167767:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5110.325367] Lustre: lustre-MDT0000: Recovery over after 0:21, of 2 clients 0 recovered and 2 were evicted. [ 5111.062658] Lustre: server umount lustre-MDT0000 complete [ 5113.315209] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5113.337761] LustreError: 167745:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5113.359234] LustreError: 167745:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 5115.228073] LustreError: 164661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784190832 with bad export cookie 2349095573911993509 [ 5115.241742] LustreError: 163990:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5115.248354] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5115.256053] LustreError: 164661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5115.900590] Lustre: server umount lustre-MDT0001 complete [ 5128.810729] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 5131.763648] Key type lgssc unregistered [ 5132.110879] LNet: 169170:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5132.124710] LNetError: 169170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5132.162762] LNet: Removed LNI 192.168.203.142@tcp [ 5133.330058] Key type .llcrypt unregistered [ 5133.334814] Key type ._llcrypt unregistered [ 5158.272673] Key type ._llcrypt registered [ 5158.278302] Key type .llcrypt registered [ 5158.465949] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 5160.372624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5160.437579] alg: No test for adler32 (adler32-zlib) [ 5161.717524] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 5162.145403] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 5163.856462] Key type lgssc registered [ 5165.002863] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5174.375249] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5174.395207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5175.972620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5180.776764] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5185.524925] LustreError: 170585:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5185.546729] LustreError: 170585:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5189.455420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5189.744254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5194.352562] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5203.023339] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5208.871325] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5215.791608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5216.185284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5220.271035] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5222.072111] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5229.273115] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5235.683501] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5235.698376] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5235.726546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5239.781027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5239.786081] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5239.806354] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5241.330874] Lustre: server umount lustre-OST0000 complete [ 5245.923270] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5245.930607] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5245.960520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5245.966194] Lustre: Skipped 1 previous similar message [ 5250.018517] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5250.020321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5250.047893] Lustre: Skipped 1 previous similar message [ 5251.753399] Lustre: server umount lustre-MDT0000 complete [ 5255.138423] LustreError: 170582:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5255.164795] LustreError: 170582:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5255.663437] LustreError: 170566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784190972 with bad export cookie 141157328746702393 [ 5255.679953] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5255.681411] LustreError: 170566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5256.364940] Lustre: server umount lustre-MDT0001 complete [ 5267.263931] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 5269.789342] Key type lgssc unregistered [ 5270.068314] LNet: 174118:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5270.072994] LNetError: 174118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5271.149606] LNet: Removed LNI 192.168.203.142@tcp [ 5271.942227] Key type .llcrypt unregistered [ 5271.945106] Key type ._llcrypt unregistered [ 5302.702631] Key type ._llcrypt registered [ 5302.710844] Key type .llcrypt registered [ 5302.848555] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 5303.941250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5304.109222] alg: No test for adler32 (adler32-zlib) [ 5305.115756] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 5305.366711] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 5307.114123] Key type lgssc registered [ 5308.532821] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5315.842250] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5315.895820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5317.651753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5322.267100] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5327.842436] LustreError: 175820:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5327.862420] LustreError: 175820:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5328.895082] LustreError: 175821:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5332.489093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5333.105227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5337.678268] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5344.721356] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5350.459622] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5357.411756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5357.830090] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5363.958180] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5364.314037] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5371.922204] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5386.213764] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5386.217886] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5386.236336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5387.747275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5387.750707] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5387.759157] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5391.253983] Lustre: server umount lustre-OST0000 complete [ 5397.994265] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5398.007913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5398.010752] Lustre: Skipped 1 previous similar message [ 5400.885617] Lustre: server umount lustre-MDT0000 complete [ 5403.109514] LustreError: 175821:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5403.133251] LustreError: 175821:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5404.910974] LustreError: 178421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784191121 with bad export cookie 17544420105479452263 [ 5404.914119] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5404.934500] LustreError: 178421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5405.327921] Lustre: server umount lustre-MDT0001 complete [ 5416.367889] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 5419.130930] Key type lgssc unregistered [ 5419.439920] LNet: 179373:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5419.449154] LNetError: 179373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5419.465217] LNet: Removed LNI 192.168.203.142@tcp [ 5420.289159] Key type .llcrypt unregistered [ 5420.293493] Key type ._llcrypt unregistered [ 5439.425604] Key type ._llcrypt registered [ 5439.428623] Key type .llcrypt registered [ 5439.510803] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 5440.314040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5440.430994] alg: No test for adler32 (adler32-zlib) [ 5441.537617] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 5441.734462] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 5443.394497] Key type lgssc registered [ 5444.594214] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5456.502910] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5456.538178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5458.039767] Lustre: *** cfs_fail_loc=135, val=0*** [ 5458.159793] LustreError: 180876:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5458.167881] LustreError: 180876:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.142@tcp: cfg command failed: rc = -2 [ 5458.176044] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5458.176044] [ 5458.184049] LustreError: MGC192.168.203.142@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5458.197196] LustreError: 180837:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5458.204657] LustreError: 180837:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5458.210942] LustreError: 180837:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5458.219942] LustreError: 180837:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5458.318253] Lustre: server umount lustre-MDT0000 complete [ 5458.320961] LustreError: 180837:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5464.489803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5464.963559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5469.776456] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5474.282395] LustreError: 181248:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5474.319543] LustreError: 181248:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5478.343279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5478.797203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5483.156322] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5490.455699] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5496.197928] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5498.407716] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5498.411057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5498.419920] Lustre: Skipped 1 previous similar message [ 5498.439132] Lustre: Skipped 1 previous similar message [ 5503.459729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5503.470441] Lustre: Skipped 1 previous similar message [ 5504.483568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5504.490778] Lustre: Skipped 1 previous similar message [ 5504.763517] Lustre: server umount lustre-MDT0000 complete [ 5508.675917] LustreError: 181229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784191225 with bad export cookie 17009726040428307556 [ 5508.683665] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5508.687681] LustreError: 181229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5508.939707] Lustre: server umount lustre-MDT0001 complete [ 5520.902771] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 5523.969395] Key type lgssc unregistered [ 5524.306515] LNet: 183566:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5524.313759] LNetError: 183566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5524.348671] LNet: Removed LNI 192.168.203.142@tcp [ 5525.112580] Key type .llcrypt unregistered [ 5525.114607] Key type ._llcrypt unregistered [ 5548.311866] Key type ._llcrypt registered [ 5548.317186] Key type .llcrypt registered [ 5548.497118] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 5548.903454] Key type .llcrypt unregistered [ 5548.908804] Key type ._llcrypt unregistered [ 5566.058911] Key type ._llcrypt registered [ 5566.061638] Key type .llcrypt registered [ 5566.138717] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 5567.127132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5567.184662] alg: No test for adler32 (adler32-zlib) [ 5568.280680] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 5568.600056] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 5570.408181] Key type lgssc registered [ 5572.003993] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5579.209187] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5579.248423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5594.786062] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5599.904384] LustreError: 185697:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.142@tcp: failed processing log, type 4: rc = -110 [ 5626.592161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5633.187144] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5641.253374] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5648.097185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5648.190801] LustreError: 185033:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d04fa4fb x1870860160862464/t0(0) o250->MGC192.168.203.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5648.682627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5648.753070] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5653.988646] LustreError: 185723:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5654.023305] LustreError: 185723:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5655.308257] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5659.116454] LustreError: 185722:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5661.159667] LustreError: 185723:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5661.200610] LustreError: 185723:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5666.274114] LustreError: 186736:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5666.301154] LustreError: 186736:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5668.373636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5668.883361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5674.748676] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5683.758981] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5689.669187] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5693.205243] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5704.678621] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5704.684827] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5704.690968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5704.699780] Lustre: Skipped 1 previous similar message [ 5709.803683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5709.825866] Lustre: Skipped 1 previous similar message [ 5711.127295] Lustre: server umount lustre-OST0000 complete [ 5720.050534] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5720.051738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5720.057797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5720.062376] Lustre: Skipped 2 previous similar messages [ 5721.946553] Lustre: server umount lustre-MDT0000 complete [ 5725.157396] LustreError: 186734:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5725.195124] LustreError: 186734:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5725.880727] LustreError: 185705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784191442 with bad export cookie 7998953665697625784 [ 5725.884105] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5725.902209] LustreError: 185705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5726.241338] Lustre: server umount lustre-MDT0001 complete [ 5737.072240] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 5739.394250] Key type lgssc unregistered [ 5739.654404] LNet: 189663:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5739.671720] LNetError: 189663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5739.692039] LNet: Removed LNI 192.168.203.142@tcp [ 5740.611302] Key type .llcrypt unregistered [ 5740.617820] Key type ._llcrypt unregistered [ 5763.324307] Key type ._llcrypt registered [ 5763.328850] Key type .llcrypt registered [ 5763.494750] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 5764.715924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5764.754761] alg: No test for adler32 (adler32-zlib) [ 5765.804742] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 5766.040842] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 5767.768453] Key type lgssc registered [ 5768.698948] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5775.645466] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5775.664848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5777.101426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5780.983824] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5787.617707] LustreError: 191080:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5787.634760] LustreError: 191080:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5789.341129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5789.728348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5793.607177] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5800.603553] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5805.568892] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5813.096529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5813.401802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5819.425878] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5822.658896] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5827.013623] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5831.507637] Lustre: Failing over lustre-MDT0000 [ 5831.961879] Lustre: server umount lustre-MDT0000 complete [ 5833.186250] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5833.187391] LustreError: 191807:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5833.198464] Lustre: Skipped 1 previous similar message [ 5838.306761] LustreError: 191080:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5838.343564] LustreError: 191080:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5843.426869] LustreError: 191079:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5843.450711] LustreError: 191079:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5848.547699] LustreError: 191807:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5848.562081] LustreError: 191807:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5849.577547] Lustre: 190393:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784191550/real 1784191550] req@0000000046e5acc9 x1870860368503552/t0(0) o400->MGC192.168.203.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1784191566 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5849.615986] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5850.792609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5859.327052] LustreError: 191075:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5859.342807] LustreError: 191075:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 5859.813575] LustreError: 190391:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000044b3d730 x1870860368509696/t0(0) o250->MGC192.168.203.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5860.162474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5860.217721] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5863.909708] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5864.441881] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5865.475093] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5865.496862] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5865.591778] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5868.172645] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5877.012754] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5885.923284] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5885.947617] Lustre: Skipped 1 previous similar message [ 5885.952639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5885.956918] Lustre: Skipped 1 previous similar message [ 5886.691262] Lustre: server umount lustre-OST0000 complete [ 5891.041575] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5891.047762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5891.051355] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5891.059293] Lustre: Skipped 2 previous similar messages [ 5891.075381] Lustre: Skipped 1 previous similar message [ 5896.162443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5896.172905] Lustre: Skipped 1 previous similar message [ 5896.459638] LustreError: 195263:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (0000000065d2f397) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 5896.566741] Lustre: server umount lustre-MDT0000 complete [ 5900.186023] LustreError: 191062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784191617 with bad export cookie 18345362254868324734 [ 5900.188827] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5900.211420] LustreError: 191062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5900.450834] Lustre: server umount lustre-MDT0001 complete [ 5909.261185] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 5911.719955] Key type lgssc unregistered [ 5911.970645] LNet: 196015:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5911.979684] LNetError: 196015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5911.992620] LNet: Removed LNI 192.168.203.142@tcp [ 5912.796176] Key type .llcrypt unregistered [ 5912.800549] Key type ._llcrypt unregistered [ 5934.992951] Key type ._llcrypt registered [ 5934.994651] Key type .llcrypt registered [ 5935.089811] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 5936.084769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5936.121215] alg: No test for adler32 (adler32-zlib) [ 5937.138409] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 5937.311612] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 5939.016168] Key type lgssc registered [ 5940.470820] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5949.824121] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5949.851497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5951.424915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5956.426226] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5961.698302] LustreError: 197428:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5961.720209] LustreError: 197428:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5966.450738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5966.706498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5971.843678] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5978.941081] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5984.994725] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5991.861171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5992.215048] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5996.352112] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5998.831464] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6005.475237] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6008.202769] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6014.627689] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6028.124752] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6037.485905] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6037.488138] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6037.528373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6037.534189] Lustre: Skipped 1 previous similar message [ 6040.443266] Lustre: server umount lustre-OST0000 complete [ 6047.722694] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6047.727627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6047.739553] Lustre: Skipped 2 previous similar messages [ 6047.747875] Lustre: Skipped 1 previous similar message [ 6049.990238] Lustre: server umount lustre-MDT0000 complete [ 6052.834258] LustreError: 198158:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6052.858826] LustreError: 198158:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6053.320921] LustreError: 199964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784191770 with bad export cookie 8688075045915958829 [ 6053.322624] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6053.330441] LustreError: 199964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6053.754421] Lustre: server umount lustre-MDT0001 complete [ 6064.958376] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 6068.339659] Key type lgssc unregistered [ 6068.611887] LNet: 201279:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6068.622872] LNetError: 201279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6068.669572] LNet: Removed LNI 192.168.203.142@tcp [ 6069.625108] Key type .llcrypt unregistered [ 6069.629238] Key type ._llcrypt unregistered [ 6087.469319] Key type ._llcrypt registered [ 6087.471442] Key type .llcrypt registered [ 6087.590985] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 6088.376900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6088.509341] alg: No test for adler32 (adler32-zlib) [ 6089.617249] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 6089.874156] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 6091.584322] Key type lgssc registered [ 6092.643756] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6099.377910] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6099.390840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6100.816972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6104.329642] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6110.181970] LustreError: 202546:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6110.222769] LustreError: 202546:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6112.423794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6112.719895] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6116.855451] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6124.623768] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6129.271697] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6135.469142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6135.816626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6141.417153] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6146.103818] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6148.732782] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6154.481344] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6165.466566] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6173.384221] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6178.228400] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6192.097478] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6192.111445] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6192.129582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6192.137932] Lustre: Skipped 1 previous similar message [ 6192.153626] Lustre: Skipped 1 previous similar message [ 6197.233468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6197.252653] Lustre: Skipped 1 previous similar message [ 6202.339518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6202.348620] Lustre: Skipped 1 previous similar message [ 6204.384636] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6204.568698] Lustre: server umount lustre-OST0000 complete [ 6210.028537] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6210.043990] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6210.059864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6212.583770] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6214.597236] Lustre: server umount lustre-MDT0000 complete [ 6217.700205] LustreError: 202543:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6217.725684] LustreError: 202543:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6218.503886] LustreError: 206768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784191935 with bad export cookie 712086026677402921 [ 6218.512396] LustreError: 206768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6218.512885] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6218.921377] Lustre: server umount lustre-MDT0001 complete [ 6229.701561] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 6232.185843] Key type lgssc unregistered [ 6232.466514] LNet: 207521:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6232.480057] LNetError: 207521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6232.497848] LNet: Removed LNI 192.168.203.142@tcp [ 6233.398602] Key type .llcrypt unregistered [ 6233.408403] Key type ._llcrypt unregistered [ 6256.326055] Key type ._llcrypt registered [ 6256.331889] Key type .llcrypt registered [ 6256.411586] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 6257.204359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6257.332235] alg: No test for adler32 (adler32-zlib) [ 6258.322723] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 6258.567043] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 6260.216106] Key type lgssc registered [ 6261.111818] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6268.136398] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6268.156431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6269.637351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6273.872844] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6279.150503] LustreError: 208933:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6279.166491] LustreError: 208933:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6283.818555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6284.291538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6289.318476] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6296.502573] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6302.536354] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6309.754551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6310.203068] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6314.299556] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6315.696671] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6323.167420] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6334.949651] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6334.956932] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6334.958743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6334.961278] Lustre: Skipped 1 previous similar message [ 6334.983977] LustreError: Skipped 1 previous similar message [ 6337.035222] Lustre: server umount lustre-OST0000 complete [ 6340.587958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6340.601930] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6340.618094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6340.626373] Lustre: Skipped 1 previous similar message [ 6345.195781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6345.208213] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6346.651941] Lustre: server umount lustre-MDT0000 complete [ 6350.316620] LustreError: 208927:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6350.357693] LustreError: 208927:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6350.784216] LustreError: 208913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784192067 with bad export cookie 18173038626609252043 [ 6350.790111] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6350.794934] LustreError: 208913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6351.095829] Lustre: server umount lustre-MDT0001 complete [ 6362.085776] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 6364.891533] Key type lgssc unregistered [ 6365.240417] LNet: 212614:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6365.251416] LNetError: 212614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6365.269211] LNet: Removed LNI 192.168.203.142@tcp [ 6366.106328] Key type .llcrypt unregistered [ 6366.111601] Key type ._llcrypt unregistered [ 6385.389467] Key type ._llcrypt registered [ 6385.397451] Key type .llcrypt registered [ 6385.525933] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 6386.397885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6386.552106] alg: No test for adler32 (adler32-zlib) [ 6387.622350] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 6387.929946] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 6389.624234] Key type lgssc registered [ 6390.672639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6397.402734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6397.427413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6398.837972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6402.796377] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6408.168541] LustreError: 213882:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6408.196790] LustreError: 213882:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6410.798020] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6411.115309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6415.636251] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6422.389188] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6427.740436] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6434.272432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6434.576815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6441.419265] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6443.837650] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6448.487600] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6464.482140] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6464.499481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6464.503319] Lustre: Skipped 1 previous similar message [ 6468.829753] Lustre: server umount lustre-OST0000 complete [ 6472.673428] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6472.684400] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6472.707320] Lustre: Skipped 1 previous similar message [ 6472.714242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6474.727692] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6474.734448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6474.767587] Lustre: Skipped 1 previous similar message [ 6478.744090] Lustre: server umount lustre-MDT0000 complete [ 6479.847419] LustreError: 213882:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6479.865461] LustreError: 213882:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6482.959910] LustreError: 213863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784192199 with bad export cookie 108156693996812619 [ 6482.966695] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6482.973825] LustreError: 213863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6483.272528] Lustre: server umount lustre-MDT0001 complete [ 6496.826833] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 6500.432812] Key type lgssc unregistered [ 6500.792562] LNet: 217565:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6500.802194] LNetError: 217565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6501.870801] LNet: Removed LNI 192.168.203.142@tcp [ 6503.081485] Key type .llcrypt unregistered [ 6503.084323] Key type ._llcrypt unregistered [ 6521.220294] Key type ._llcrypt registered [ 6521.223192] Key type .llcrypt registered [ 6521.320105] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 6522.300914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6522.364113] alg: No test for adler32 (adler32-zlib) [ 6523.517673] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 6523.734267] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 6525.394035] Key type lgssc registered [ 6526.556393] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6535.140777] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6535.155330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6536.485146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6540.782716] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6546.918027] LustreError: 218829:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6546.929893] LustreError: 218829:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6548.373277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6548.685528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6553.062628] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6560.945624] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6567.118851] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6573.442194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6573.551362] LustreError: 220648:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6573.924657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6580.927785] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6581.244128] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6588.947530] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6598.024175] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6598.088634] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6598.365080] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6604.889888] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6611.887280] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6613.584702] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6625.337707] Lustre: Permanently deactivating lustre-OST0001 [ 6625.352991] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6634.054223] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6645.021950] Lustre: Permanently reactivating lustre-OST0001 [ 6645.057154] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6645.064756] Lustre: Skipped 2 previous similar messages [ 6650.955989] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6650.969521] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6650.977840] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6650.988814] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6651.057967] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6656.022856] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6656.045961] Lustre: Skipped 1 previous similar message [ 6656.051415] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6656.059567] Lustre: Skipped 1 previous similar message [ 6656.878327] Lustre: server umount lustre-OST0001 complete [ 6661.091106] LustreError: 220651:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6661.107069] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6661.108573] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6661.108581] Lustre: Skipped 1 previous similar message [ 6661.110087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6661.113704] LustreError: 220651:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6666.212976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6666.214306] LustreError: 222980:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6666.223380] Lustre: Skipped 2 previous similar messages [ 6666.239953] LustreError: 222980:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6666.861539] Lustre: server umount lustre-OST0000 complete [ 6671.333077] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6671.357259] Lustre: Skipped 1 previous similar message [ 6671.360178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6676.456296] LustreError: 219557:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6676.483929] LustreError: 219557:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6676.578972] Lustre: server umount lustre-MDT0000 complete [ 6680.122959] LustreError: 218810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784192397 with bad export cookie 6480979411171844062 [ 6680.125591] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6680.138305] LustreError: 218810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6680.449859] Lustre: server umount lustre-MDT0001 complete [ 6690.451880] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 6693.007829] Key type lgssc unregistered [ 6693.303884] LNet: 223932:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6693.314399] LNetError: 223932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6694.378810] LNet: Removed LNI 192.168.203.142@tcp [ 6695.213481] Key type .llcrypt unregistered [ 6695.215677] Key type ._llcrypt unregistered [ 6735.455135] Key type ._llcrypt registered [ 6735.459607] Key type .llcrypt registered [ 6735.580723] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 6736.942553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6736.995425] alg: No test for adler32 (adler32-zlib) [ 6737.992629] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 6738.240844] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 6740.091148] Key type lgssc registered [ 6741.263816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6749.794929] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6749.823240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6751.144624] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6751.174459] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6751.416176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6755.502767] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6764.909241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6765.085788] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6765.119871] Lustre: 226148:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6765.539209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6770.263712] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6777.193693] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6782.803666] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6789.470561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6789.682233] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6789.939277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6796.495439] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6796.906266] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6805.688074] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6808.344611] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6817.081281] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6824.551676] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6830.233208] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6847.103854] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6847.114895] Lustre: Skipped 1 previous similar message [ 6869.292040] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6869.303775] Lustre: Skipped 2 previous similar messages [ 6904.623956] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6904.631042] Lustre: Skipped 4 previous similar messages [ 6975.970595] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6975.976740] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6975.986424] LustreError: Skipped 1 previous similar message [ 6975.987676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6975.996315] Lustre: Skipped 1 previous similar message [ 6976.027225] Lustre: Skipped 1 previous similar message [ 6981.095969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6981.111319] Lustre: Skipped 1 previous similar message [ 6986.218457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6986.230664] Lustre: Skipped 1 previous similar message [ 6988.258046] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6988.450613] Lustre: server umount lustre-OST0000 complete [ 6995.938121] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6995.950227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6995.967428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7000.785971] Lustre: server umount lustre-MDT0000 complete [ 7003.932541] LustreError: 229772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784192720 with bad export cookie 9605058240858954445 [ 7003.938469] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7003.945305] LustreError: 229772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7004.206026] Lustre: server umount lustre-MDT0001 complete [ 7012.813776] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 7015.062108] Key type lgssc unregistered [ 7015.297812] LNet: 230944:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7015.304127] LNetError: 230944:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7015.321212] LNet: Removed LNI 192.168.203.142@tcp [ 7016.111158] Key type .llcrypt unregistered [ 7016.113334] Key type ._llcrypt unregistered [ 7038.764262] Key type ._llcrypt registered [ 7038.772502] Key type .llcrypt registered [ 7038.871788] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 7039.842852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7039.922162] alg: No test for adler32 (adler32-zlib) [ 7041.076497] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 7041.371757] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 7043.048169] Key type lgssc registered [ 7044.239179] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7051.566236] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7051.588133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7052.999381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7057.118348] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7062.504331] LustreError: 232371:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7062.521725] LustreError: 232371:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7065.144857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7065.476860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7069.633828] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7076.392361] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7082.371603] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7088.919540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7089.352509] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7095.093756] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7097.516824] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7102.176505] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7121.378178] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7121.382054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7121.406687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7121.442648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7121.455558] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.135@tcp (at 192.168.203.135@tcp) [ 7122.307510] Lustre: server umount lustre-OST0000 complete [ 7126.497537] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7126.511459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7126.517561] Lustre: Skipped 2 previous similar messages [ 7126.536199] Lustre: Skipped 2 previous similar messages [ 7131.620783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7131.627428] Lustre: Skipped 1 previous similar message [ 7131.761623] Lustre: server umount lustre-MDT0000 complete [ 7134.756417] LustreError: 234216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784192851 with bad export cookie 9425617406921388186 [ 7134.760632] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7134.763844] LustreError: 234216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7135.023323] Lustre: server umount lustre-MDT0001 complete [ 7144.377243] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 7147.011170] Key type lgssc unregistered [ 7147.311980] LNet: 236066:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7147.323707] LNetError: 236066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7147.342630] LNet: Removed LNI 192.168.203.142@tcp [ 7148.131907] Key type .llcrypt unregistered [ 7148.139026] Key type ._llcrypt unregistered [ 7169.061870] Key type ._llcrypt registered [ 7169.065027] Key type .llcrypt registered [ 7169.170914] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 7170.234654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7170.266070] alg: No test for adler32 (adler32-zlib) [ 7171.316607] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 7171.581603] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 7173.281222] Key type lgssc registered [ 7174.414107] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7447.615345] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 7451.125928] Key type lgssc unregistered [ 7451.813912] LNet: 238089:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7451.835328] LNetError: 238089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7451.850845] LNet: Removed LNI 192.168.203.142@tcp [ 7452.718440] Key type .llcrypt unregistered [ 7452.723266] Key type ._llcrypt unregistered [ 7487.700516] Key type ._llcrypt registered [ 7487.702245] Key type .llcrypt registered [ 7487.800248] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 7504.098418] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 7505.215302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7505.232512] alg: No test for adler32 (adler32-zlib) [ 7506.268979] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 7506.466676] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 7508.120305] Key type lgssc registered [ 7509.103327] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7536.206051] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7227 sec ========== 05:14:11 (1784193251) [ 7537.784505] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 05:14:12 (1784193252) === [ 7541.020861] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 05:14:15 (1784193255) === [ 7619.168480] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 7621.312863] Key type lgssc unregistered [ 7621.537520] LNet: 243346:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7621.551753] LNetError: 243346:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7621.571732] LNet: Removed LNI 192.168.203.142@tcp [ 7622.188670] Key type .llcrypt unregistered [ 7622.190786] Key type ._llcrypt unregistered