[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 431863286 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002362] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.007349] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010100] LSM: Security Framework initializing [ 0.011036] Yama: becoming mindful. [ 0.012035] SELinux: Initializing. [ 0.013079] *** VALIDATE selinux *** [ 0.021603] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025779] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028140] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029103] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030102] *** VALIDATE tmpfs *** [ 0.032153] *** VALIDATE proc *** [ 0.033206] *** VALIDATE cgroup *** [ 0.034007] *** VALIDATE cgroup2 *** [ 0.035242] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036134] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038028] Spectre V2 : User space: Vulnerable [ 0.039009] Speculative Store Bypass: Vulnerable [ 0.042197] debug: unmapping init [mem 0xffffffffaa459000-0xffffffffaa460fff] [ 0.044842] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045643] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046022] ... version: 2 [ 0.047010] ... bit width: 48 [ 0.048010] ... generic registers: 4 [ 0.049010] ... value mask: 0000ffffffffffff [ 0.050010] ... max period: 00007fffffffffff [ 0.051012] ... fixed-purpose events: 3 [ 0.052010] ... event mask: 000000070000000f [ 0.054202] rcu: Hierarchical SRCU implementation. [ 0.056410] smp: Bringing up secondary CPUs ... [ 0.057599] x86: Booting SMP configuration: [ 0.058025] .... node #0, CPUs: #1 #2 #3 [ 0.062024] smp: Brought up 1 node, 4 CPUs [ 0.064014] smpboot: Max logical packages: 1 [ 0.065014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.273088] node 0 deferred pages initialised in 206ms [ 0.277212] devtmpfs: initialized [ 0.278247] x86/mm: Memory block size: 128MB [ 0.280557] gcov: version magic: 0x41383552 [ 0.283285] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.284084] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.286357] pinctrl core: initialized pinctrl subsystem [ 0.288144] [ 0.288562] ************************************************************* [ 0.291016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.293012] ** ** [ 0.295014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.297015] ** ** [ 0.300019] ** This means that this kernel is built to expose internal ** [ 0.302013] ** IOMMU data structures, which may compromise security on ** [ 0.304019] ** your system. ** [ 0.307018] ** ** [ 0.309013] ** If you see this message and you are not debugging the ** [ 0.311021] ** kernel, report this immediately to your vendor! ** [ 0.314026] ** ** [ 0.316015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.318014] ************************************************************* [ 0.321987] NET: Registered protocol family 16 [ 0.324536] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.327082] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.331080] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.335037] cpuidle: using governor menu [ 0.337901] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.339515] PCI: Using configuration type 1 for base access [ 0.342143] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.349071] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.350028] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.353130] cryptd: max_cpu_qlen set to 1000 [ 0.356323] ACPI: Added _OSI(Module Device) [ 0.358015] ACPI: Added _OSI(Processor Device) [ 0.359014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.361015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.365802] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.371391] ACPI: Interpreter enabled [ 0.373056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.374012] ACPI: Using IOAPIC for interrupt routing [ 0.376119] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.378482] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.388935] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.391031] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.393029] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.397079] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.402599] acpiphp: Slot [2] registered [ 0.404149] acpiphp: Slot [5] registered [ 0.405126] acpiphp: Slot [6] registered [ 0.406081] acpiphp: Slot [7] registered [ 0.407086] acpiphp: Slot [8] registered [ 0.409179] acpiphp: Slot [9] registered [ 0.410134] acpiphp: Slot [10] registered [ 0.412126] acpiphp: Slot [3] registered [ 0.413085] acpiphp: Slot [4] registered [ 0.415104] acpiphp: Slot [11] registered [ 0.416075] acpiphp: Slot [12] registered [ 0.417076] acpiphp: Slot [13] registered [ 0.418073] acpiphp: Slot [14] registered [ 0.420070] acpiphp: Slot [15] registered [ 0.421087] acpiphp: Slot [16] registered [ 0.423157] acpiphp: Slot [17] registered [ 0.424081] acpiphp: Slot [18] registered [ 0.425113] acpiphp: Slot [19] registered [ 0.427102] acpiphp: Slot [20] registered [ 0.428080] acpiphp: Slot [21] registered [ 0.429078] acpiphp: Slot [22] registered [ 0.431083] acpiphp: Slot [23] registered [ 0.432133] acpiphp: Slot [24] registered [ 0.433075] acpiphp: Slot [25] registered [ 0.435071] acpiphp: Slot [26] registered [ 0.436118] acpiphp: Slot [27] registered [ 0.437077] acpiphp: Slot [28] registered [ 0.438083] acpiphp: Slot [29] registered [ 0.440098] acpiphp: Slot [30] registered [ 0.441079] acpiphp: Slot [31] registered [ 0.442049] PCI host bridge to bus 0000:00 [ 0.444015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.446017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.448022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.450027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.453024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.455016] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.457203] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.459920] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.462317] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.471014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.475891] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.478012] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.480026] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.483015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.484451] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.488086] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.491051] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.493781] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.498013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.510023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.513017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.518217] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.522010] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.528021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.542015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.551724] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.560020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.566018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.586017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.596311] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.601020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.609016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.623021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.634715] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.642018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.648020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.666019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.676090] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.682021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.687029] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.705021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.713856] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.719017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.724022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.745017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.755274] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.757255] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.759328] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.762688] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.764192] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.768170] iommu: Default domain type: Passthrough [ 0.770254] SCSI subsystem initialized [ 0.771116] ACPI: bus type USB registered [ 0.773072] usbcore: registered new interface driver usbfs [ 0.775059] usbcore: registered new interface driver hub [ 0.776057] usbcore: registered new device driver usb [ 0.778157] pps_core: LinuxPPS API ver. 1 registered [ 0.779008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.782073] PTP clock support registered [ 0.783129] EDAC MC: Ver: 3.0.0 [ 0.785098] PCI: Using ACPI for IRQ routing [ 0.786740] NetLabel: Initializing [ 0.788008] NetLabel: domain hash size = 128 [ 0.789008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.790073] NetLabel: unlabeled traffic allowed by default [ 0.792182] vgaarb: loaded [ 0.793275] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.795015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.801345] clocksource: Switched to clocksource kvm-clock [ 0.906291] VFS: Disk quotas dquot_6.6.0 [ 0.907933] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.910266] *** VALIDATE ramfs *** [ 0.911673] *** VALIDATE hugetlbfs *** [ 0.913387] pnp: PnP ACPI init [ 0.916128] pnp: PnP ACPI: found 6 devices [ 0.930881] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.933454] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.935039] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.936733] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.939031] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.941545] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.943662] NET: Registered protocol family 2 [ 0.945695] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.949661] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.952414] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.956541] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.959151] TCP: Hash tables configured (established 65536 bind 65536) [ 0.961725] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.965465] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.967985] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.971326] NET: Registered protocol family 1 [ 0.973212] RPC: Registered named UNIX socket transport module. [ 0.975432] RPC: Registered udp transport module. [ 0.976865] RPC: Registered tcp transport module. [ 0.978205] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.980113] NET: Registered protocol family 44 [ 0.981474] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.983348] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.984924] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.986681] PCI: CLS 0 bytes, default 64 [ 0.987987] Unpacking initramfs... [ 2.338584] debug: unmapping init [mem 0xffff8ab63cc54000-0xffff8ab63ffbffff] [ 2.342735] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.345105] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.348203] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.841022] Initialise system trusted keyrings [ 2.842502] Key type blacklist registered [ 2.844214] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.852230] zbud: loaded [ 2.854772] *** VALIDATE nfs *** [ 2.855936] *** VALIDATE nfs4 *** [ 2.857480] pstore: using deflate compression [ 2.861473] Platform Keyring initialized [ 2.959540] NET: Registered protocol family 38 [ 2.961146] Key type asymmetric registered [ 2.962936] Asymmetric key parser 'x509' registered [ 2.965037] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.968436] io scheduler mq-deadline registered [ 2.969627] io scheduler kyber registered [ 2.971257] io scheduler bfq registered [ 2.973211] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.976308] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.979066] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.981575] ACPI: Power Button [PWRF] [ 2.986060] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.992690] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.007505] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.013254] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.028259] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.055256] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.084648] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.091147] Non-volatile memory driver v1.3 [ 3.093055] Linux agpgart interface v0.103 [ 3.129601] virtio_blk virtio1: [vda] 136464 512-byte logical blocks (69.9 MB/66.6 MiB) [ 3.134213] vda: detected capacity change from 0 to 69869568 [ 3.149692] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.152851] vdb: detected capacity change from 0 to 1073741824 [ 3.166740] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.169804] vdc: detected capacity change from 0 to 2621440000 [ 3.184850] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.187970] vdd: detected capacity change from 0 to 2621440000 [ 3.201750] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.204746] vde: detected capacity change from 0 to 4294967296 [ 3.218607] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.221216] vdf: detected capacity change from 0 to 4294967296 [ 3.228059] libphy: Fixed MDIO Bus: probed [ 3.232643] usbcore: registered new interface driver usbserial_generic [ 3.235587] usbserial: USB Serial support registered for generic [ 3.238015] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.242553] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.243873] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.247186] mousedev: PS/2 mouse device common for all mice [ 3.250016] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.251747] rtc_cmos 00:05: RTC can wake from S4 [ 3.256129] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.256508] rtc_cmos 00:05: registered as rtc0 [ 3.260968] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.261123] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.266198] intel_pstate: CPU model not supported [ 3.268946] hid: raw HID events driver (C) Jiri Kosina [ 3.270848] usbcore: registered new interface driver usbhid [ 3.273221] usbhid: USB HID core driver [ 3.274939] drop_monitor: Initializing network drop monitor service [ 3.277303] Initializing XFRM netlink socket [ 3.279054] NET: Registered protocol family 10 [ 3.282913] Segment Routing with IPv6 [ 3.284445] NET: Registered protocol family 17 [ 3.286343] mpls_gso: MPLS GSO support [ 3.291732] RAS: Correctable Errors collector initialized. [ 3.294877] AVX version of gcm_enc/dec engaged. [ 3.297107] AES CTR mode by8 optimization enabled [ 3.361459] sched_clock: Marking stable (3361440936, 0)->(4188934454, -827493518) [ 3.364507] registered taskstats version 1 [ 3.366084] Loading compiled-in X.509 certificates [ 3.368230] zswap: loaded using pool lzo/zbud [ 3.392098] Key type big_key registered [ 3.403150] Key type encrypted registered [ 3.404697] ima: No TPM chip found, activating TPM-bypass! [ 3.406843] ima: Allocated hash algorithm: sha1 [ 3.408533] ima: No architecture policies found [ 3.409919] evm: Initialising EVM extended attributes: [ 3.411692] evm: security.selinux [ 3.412668] evm: security.ima [ 3.415210] evm: security.capability [ 3.416666] evm: HMAC attrs: 0x1 [ 3.419138] rtc_cmos 00:05: setting system clock to 2026-06-12 02:02:27 UTC (1781229747) [ 3.424655] debug: unmapping init [mem 0xffffffffab403000-0xffffffffab5fffff] [ 3.427484] debug: unmapping init [mem 0xffffffffaa182000-0xffffffffaa458fff] [ 3.437140] Write protecting the kernel read-only data: 28672k [ 3.439861] debug: unmapping init [mem 0xffffffffa8803000-0xffffffffa89fffff] [ 3.442707] debug: unmapping init [mem 0xffffffffa9114000-0xffffffffa91fffff] [ 3.478687] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.487147] systemd[1]: Detected virtualization kvm. [ 3.488896] systemd[1]: Detected architecture x86-64. [ 3.490691] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.516335] systemd[1]: No hostname configured. [ 3.517808] systemd[1]: Set hostname to . [ 3.519558] random: systemd: uninitialized urandom read (16 bytes read) [ 3.521976] systemd[1]: Initializing machine ID from random generator. [ 3.575538] random: ln: uninitialized urandom read (6 bytes read) [ 3.655799] random: systemd: uninitialized urandom read (16 bytes read) [ 3.657712] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.662808] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.667677] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Swap. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. Starting Setup Virtual Console... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Journal Service. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.268065] device-mapper: uevent: version 1.0.3 [ 4.269701] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.960753] virtio_net virtio0 ens2: renamed from eth0 [ 4.968811] random: fast init done [ 4.997243] scsi host0: ata_piix [ 5.043961] scsi host1: ata_piix [ 5.049967] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.052385] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.342461] dracut-initqueue[589]: RTNETLINK answers: File exists [ 9.923221] random: crng init done [ 9.924609] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.200653] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.260631] printk: systemd: 21 output lines suppressed due to ratelimiting [ 11.503145] SELinux: Disabled at runtime. [ 11.558329] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.564127] systemd[1]: Detected virtualization kvm. [ 11.566031] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.012069] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.015173] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.019718] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.023479] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.026431] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.033143] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.039183] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel V[ 12.158266] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ariables... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.389419] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.741692] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.750870] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.927822] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.938161] EDAC sbridge: Ver: 1.1.2 [ 14.317134] Key type dns_resolver registered [ 14.622542] NFS: Registering the id_resolver key type [ 14.625217] Key type id_resolver registered [ 14.627553] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg646-server login: [ 39.441259] libcfs: loading out-of-tree module taints kernel. [ 39.460905] Key type ._llcrypt registered [ 39.462446] Key type .llcrypt registered [ 39.503494] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 48.143699] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 48.802757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 48.810210] alg: No test for adler32 (adler32-zlib) [ 49.821365] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 50.162979] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 51.783180] Key type lgssc registered [ 52.546639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 60.152412] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 75.472592] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 80.273676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 80.284183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 81.393943] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 81.409208] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 81.451608] Lustre: lustre-MDT0000: new disk, initializing [ 81.503917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 81.513145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 83.072111] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 89.016603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 89.060676] Lustre: 6476:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 89.077172] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 89.079559] Lustre: Skipped 1 previous similar message [ 89.120097] Lustre: lustre-MDT0001: new disk, initializing [ 89.145780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 89.158974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 89.164106] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 90.646348] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 93.121959] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 96.979289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 97.095703] Lustre: lustre-OST0000: new disk, initializing [ 97.098378] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 97.102119] Lustre: 8382:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 97.133246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 99.413532] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 101.045939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 101.052816] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 101.080395] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 106.224061] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 106.281664] Lustre: lustre-OST0001: new disk, initializing [ 106.284911] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 106.289109] Lustre: 9440:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 106.314078] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 108.725846] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 111.093569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 111.098214] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 111.113402] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 115.438926] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 118.461480] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 120.929630] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing check_logdir /tmp/testlogs/ [ 122.646851] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing yml_node [ 124.424664] Lustre: DEBUG MARKER: Client: 2.17.53.61 [ 125.448762] Lustre: DEBUG MARKER: MDS: 2.17.53.61 [ 126.448043] Lustre: DEBUG MARKER: OSS: 2.17.53.61 [ 127.080651] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jun 11 22:04:30 EDT 2026 [ 134.200267] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 134.958943] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 150.495556] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 150.498426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 150.504175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 152.032105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 152.032634] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 152.037282] Lustre: Skipped 2 previous similar messages [ 152.047814] Lustre: Skipped 2 previous similar messages [ 157.152102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 157.156620] Lustre: Skipped 4 previous similar messages [ 161.247134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 161.318523] Lustre: server umount lustre-MDT0000 complete [ 162.689291] LustreError: 10238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781229906 with bad export cookie 1567039663776279614 [ 162.690384] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 162.695902] LustreError: 10238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 167.391868] LustreError: 8390:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 167.391996] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 167.392238] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 167.401695] LustreError: 8390:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 167.407281] Lustre: Skipped 1 previous similar message [ 168.877669] Lustre: server umount lustre-MDT0001 complete [ 176.527142] Lustre: server umount lustre-OST0000 complete [ 184.223506] Lustre: server umount lustre-OST0001 complete [ 186.914144] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 190.698200] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 206.063325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 206.177096] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 206.198072] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 206.244396] Lustre: lustre-MDT0000: new disk, initializing [ 206.285691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 206.294964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 207.909295] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 213.104461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 213.178094] Lustre: 14944:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 213.336835] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 215.103670] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 218.746183] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 220.231868] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 223.116596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 223.247987] Lustre: lustre-OST0000: new disk, initializing [ 223.249847] Lustre: Skipped 1 previous similar message [ 223.252497] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 223.255417] Lustre: Skipped 2 previous similar messages [ 223.258731] Lustre: 16684:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 223.301542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 223.305174] Lustre: Skipped 1 previous similar message [ 224.968439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 224.971755] Lustre: Skipped 1 previous similar message [ 224.973956] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 225.014788] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 225.822224] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 229.549648] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 231.312197] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 231.405084] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 233.052951] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 233.146210] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 235.490804] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 235.493295] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 235.493987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 235.493991] Lustre: Skipped 1 previous similar message [ 235.498232] Lustre: Skipped 1 previous similar message [ 245.731938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 245.735540] Lustre: Skipped 4 previous similar messages [ 248.287245] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 248.369754] Lustre: server umount lustre-OST0000 complete [ 250.849460] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 250.856584] Lustre: Skipped 1 previous similar message [ 264.159146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 264.242843] Lustre: server umount lustre-MDT0000 complete [ 265.957559] LustreError: 14935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230010 with bad export cookie 1567039663776281147 [ 265.961308] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 265.964666] LustreError: 14935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 266.100792] Lustre: server umount lustre-MDT0001 complete [ 267.579330] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 22:06:50 (1781230010) [ 270.427961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 270.625487] LustreError: 18800:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 270.662426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 272.300198] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 275.938593] LustreError: 18801:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 276.244806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 278.265106] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 281.073862] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 283.019483] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 286.421758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 289.131765] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 291.464673] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 301.026768] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 301.026784] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 301.027892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 301.027898] Lustre: Skipped 6 previous similar messages [ 301.039212] Lustre: Skipped 1 previous similar message [ 306.009790] Lustre: server umount lustre-OST0000 complete [ 313.629661] Lustre: server umount lustre-MDT0000 complete [ 315.191797] LustreError: 19516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230059 with bad export cookie 1567039663776282029 [ 315.196068] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 315.197844] LustreError: 19516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 315.324708] Lustre: server umount lustre-MDT0001 complete [ 319.606507] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 320.866032] Key type lgssc unregistered [ 321.019138] LNet: 21972:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 321.023224] LNetError: 21972:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 321.031276] LNet: Removed LNI 192.168.206.146@tcp [ 321.425169] Key type .llcrypt unregistered [ 321.426730] Key type ._llcrypt unregistered [ 331.474238] Key type ._llcrypt registered [ 331.476147] Key type .llcrypt registered [ 331.529240] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 331.937936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 332.002353] alg: No test for adler32 (adler32-zlib) [ 332.900418] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 333.003298] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 334.599140] Key type lgssc registered [ 335.039575] Lustre: Echo OBD driver; http://www.lustre.org/ [ 338.297846] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 338.309449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 339.547736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 341.129980] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 344.796272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 344.972994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 346.547711] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 348.712972] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 350.290284] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 353.054023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 353.211553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 355.619221] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 357.776180] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 364.457834] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 369.635507] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 369.637841] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 369.638549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 369.641474] Lustre: Skipped 1 previous similar message [ 370.839076] Lustre: server umount lustre-OST0000 complete [ 374.360735] Lustre: server umount lustre-MDT0000 complete [ 374.753426] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 374.754442] LustreError: 23345:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 374.757772] Lustre: Skipped 1 previous similar message [ 374.768105] LustreError: 23345:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 375.831385] LustreError: 23327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230119 with bad export cookie 7725571087475415060 [ 375.834140] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 375.835597] LustreError: 23327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 375.943374] Lustre: server umount lustre-MDT0001 complete [ 380.150901] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 381.464130] Key type lgssc unregistered [ 381.642845] LNet: 26958:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 381.648462] LNetError: 26958:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 381.657543] LNet: Removed LNI 192.168.206.146@tcp [ 382.093721] Key type .llcrypt unregistered [ 382.096324] Key type ._llcrypt unregistered [ 393.147918] Key type ._llcrypt registered [ 393.150122] Key type .llcrypt registered [ 393.201751] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 393.616824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 393.649311] alg: No test for adler32 (adler32-zlib) [ 394.559625] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 394.669224] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 396.271131] Key type lgssc registered [ 396.823897] Lustre: Echo OBD driver; http://www.lustre.org/ [ 400.103969] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 400.114782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 401.323988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 402.952348] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 406.620329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 406.807356] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 408.501777] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 411.051642] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 412.653299] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 418.258350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 418.421403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 420.689089] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 422.944049] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 427.622449] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 438.241070] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 438.244886] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 438.253806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 442.337603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 442.338475] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 442.338941] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 442.341863] Lustre: Skipped 1 previous similar message [ 443.485957] Lustre: server umount lustre-OST0000 complete [ 447.457909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 447.463207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 447.466922] Lustre: Skipped 1 previous similar message [ 451.241798] Lustre: server umount lustre-MDT0000 complete [ 452.576629] LustreError: 30933:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 452.583799] LustreError: 30933:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 452.833595] LustreError: 28311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230196 with bad export cookie 14553600799109535406 [ 452.837315] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 452.840711] LustreError: 28311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 452.992279] Lustre: server umount lustre-MDT0001 complete [ 457.835775] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 459.197394] Key type lgssc unregistered [ 459.359537] LNet: 31869:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 459.363943] LNetError: 31869:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 459.372568] LNet: Removed LNI 192.168.206.146@tcp [ 459.754441] Key type .llcrypt unregistered [ 459.756184] Key type ._llcrypt unregistered [ 470.939879] Key type ._llcrypt registered [ 470.941539] Key type .llcrypt registered [ 470.985378] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 471.478709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 471.492732] alg: No test for adler32 (adler32-zlib) [ 472.411398] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 472.529921] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 474.151164] Key type lgssc registered [ 474.747404] Lustre: Echo OBD driver; http://www.lustre.org/ [ 478.432742] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 478.444718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 479.678433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 481.406607] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 485.491400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 485.731589] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 487.776923] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 490.232872] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 491.763851] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 495.068023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 495.236295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 497.744089] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 500.186519] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 504.493776] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 515.041796] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 515.046284] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 515.054614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 516.064438] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 516.065067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 516.068184] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 516.346444] Lustre: server umount lustre-OST0000 complete [ 521.185752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 521.187928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 521.196051] Lustre: Skipped 1 previous similar message [ 521.199012] Lustre: Skipped 1 previous similar message [ 524.750218] Lustre: server umount lustre-MDT0000 complete [ 526.308386] LustreError: 34350:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 526.325505] LustreError: 34350:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 526.769817] LustreError: 33223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230270 with bad export cookie 850904271651383307 [ 526.770246] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 526.778137] LustreError: 33223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 526.968073] Lustre: server umount lustre-MDT0001 complete [ 531.953780] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 533.344175] Key type lgssc unregistered [ 533.493650] LNet: 36431:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 533.497154] LNetError: 36431:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 533.508780] LNet: Removed LNI 192.168.206.146@tcp [ 533.899875] Key type .llcrypt unregistered [ 533.901747] Key type ._llcrypt unregistered [ 545.519067] Key type ._llcrypt registered [ 545.520107] Key type .llcrypt registered [ 545.563373] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 546.044562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 546.053056] alg: No test for adler32 (adler32-zlib) [ 546.945811] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 547.044807] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 548.647302] Key type lgssc registered [ 549.120744] Lustre: Echo OBD driver; http://www.lustre.org/ [ 552.865075] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 552.878228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 554.156615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 555.966533] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 559.830233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 560.002156] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 561.635089] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 563.787429] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 565.274041] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 568.085581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 568.279301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 570.697646] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 572.983203] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 579.565311] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 589.661736] Lustre: lustre-OST0000: Not available for connect from 192.168.206.46@tcp (stopping) [ 589.794284] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 589.797128] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 589.807505] Lustre: Skipped 1 previous similar message [ 593.323898] Lustre: server umount lustre-OST0000 complete [ 609.633138] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 609.644688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 609.650612] Lustre: Skipped 3 previous similar messages [ 613.795959] Lustre: server umount lustre-MDT0000 complete [ 614.880586] LustreError: 38512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 614.888493] LustreError: 38512:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 615.748934] LustreError: 39416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230359 with bad export cookie 373591807186061937 [ 615.752021] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 615.755757] LustreError: 39416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 630.239105] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 630.359568] Lustre: server umount lustre-MDT0001 complete [ 635.420863] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 636.855035] Key type lgssc unregistered [ 637.017033] LNet: 41170:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 637.021255] LNetError: 41170:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 637.034563] LNet: Removed LNI 192.168.206.146@tcp [ 637.447809] Key type .llcrypt unregistered [ 637.449813] Key type ._llcrypt unregistered [ 648.451475] Key type ._llcrypt registered [ 648.453040] Key type .llcrypt registered [ 648.508036] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 648.883745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 648.891996] alg: No test for adler32 (adler32-zlib) [ 649.798652] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 649.934693] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 651.551160] Key type lgssc registered [ 652.136689] Lustre: Echo OBD driver; http://www.lustre.org/ [ 655.978460] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 655.988054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 657.262268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 659.057616] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 663.084952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 663.292569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 665.121834] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 668.236454] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 670.351815] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 674.780716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 674.975910] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 678.260902] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 681.101288] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 682.155272] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 693.592645] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.46@tcp (stopping) [ 693.732595] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 693.743134] Lustre: Skipped 2 previous similar messages [ 695.940739] Lustre: server umount lustre-MDT0000 complete [ 697.787633] LustreError: 42526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230441 with bad export cookie 15957343281400467181 [ 697.788817] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 697.794226] LustreError: 42526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 697.823896] LustreError: 43258:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000089b9f1c1 x1867755441643776/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 698.698855] LustreError: 42543:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 698.709836] LustreError: 42543:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 698.717369] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.46@tcp (stopping) [ 698.719959] Lustre: Skipped 3 previous similar messages [ 698.848965] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 702.881522] LustreError: 42542:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 702.881845] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 702.897394] LustreError: 42542:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 702.909125] Lustre: Skipped 1 previous similar message [ 706.979052] LustreError: 44646:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 706.979756] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 706.986472] LustreError: 44646:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 706.989871] Lustre: Skipped 1 previous similar message [ 712.159554] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 712.160371] LustreError: 42541:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 712.160584] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 712.160588] Lustre: Skipped 1 previous similar message [ 712.180208] LustreError: 42541:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 712.280225] Lustre: server umount lustre-MDT0001 complete [ 724.816603] Lustre: server umount lustre-OST0000 complete [ 732.817591] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 734.617965] Key type lgssc unregistered [ 734.811150] LNet: 46047:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 734.815339] LNetError: 46047:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 734.826969] LNet: Removed LNI 192.168.206.146@tcp [ 735.237535] Key type .llcrypt unregistered [ 735.239195] Key type ._llcrypt unregistered [ 748.475607] Key type ._llcrypt registered [ 748.477289] Key type .llcrypt registered [ 748.532514] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 749.052789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 749.064710] alg: No test for adler32 (adler32-zlib) [ 749.963369] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 750.085105] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 751.695234] Key type lgssc registered [ 752.205687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 756.190108] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 756.212790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 771.743578] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 776.863295] LustreError: 47395:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.146@tcp: failed processing log, type 4: rc = -110 [ 803.487768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 806.636662] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 809.628051] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 973.391064] Lustre: server umount lustre-OST0000 complete [ 980.599859] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 981.876849] Key type lgssc unregistered [ 982.012931] LNet: 48940:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 982.016720] LNetError: 48940:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 982.025747] LNet: Removed LNI 192.168.206.146@tcp [ 982.387335] Key type .llcrypt unregistered [ 982.389114] Key type ._llcrypt unregistered [ 993.065330] Key type ._llcrypt registered [ 993.067932] Key type .llcrypt registered [ 993.116319] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 993.452608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 993.469373] alg: No test for adler32 (adler32-zlib) [ 994.347054] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 994.452369] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 996.047570] Key type lgssc registered [ 996.613756] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1000.126743] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1000.135531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1001.408642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1003.306341] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1007.330626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1007.550091] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1009.571232] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1012.283342] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1013.936326] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1016.871527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1017.040722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1019.518070] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1022.079885] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1023.457471] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1023.457815] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1023.462149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1026.530945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1029.290027] Lustre: server umount lustre-OST0000 complete [ 1031.650456] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1031.651917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1031.656019] Lustre: Skipped 1 previous similar message [ 1031.662272] Lustre: Skipped 2 previous similar messages [ 1036.768420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1036.772728] Lustre: Skipped 1 previous similar message [ 1037.385427] Lustre: server umount lustre-MDT0000 complete [ 1039.177876] LustreError: 50293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230783 with bad export cookie 2395991124393173384 [ 1039.182822] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1039.184843] LustreError: 50293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1039.348127] Lustre: server umount lustre-MDT0001 complete [ 1044.215463] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 1045.459219] Key type lgssc unregistered [ 1045.623586] LNet: 53481:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1045.628116] LNetError: 53481:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1045.637421] LNet: Removed LNI 192.168.206.146@tcp [ 1045.993726] Key type .llcrypt unregistered [ 1045.995239] Key type ._llcrypt unregistered [ 1056.233036] Key type ._llcrypt registered [ 1056.234936] Key type .llcrypt registered [ 1056.281464] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 1056.703794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1056.741102] alg: No test for adler32 (adler32-zlib) [ 1057.638843] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 1057.749621] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 1059.351240] Key type lgssc registered [ 1059.838789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1063.155826] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1063.165243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1078.623473] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1083.743216] LustreError: 54829:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.146@tcp: failed processing log, type 4: rc = -110 [ 1110.367714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1113.249125] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1115.600317] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1118.579341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1118.823165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1118.857616] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1120.577324] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1124.320958] LustreError: 54855:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1124.330430] LustreError: 54855:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1124.588950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1124.792084] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1126.723278] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1129.379303] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1130.929575] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1134.048364] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1134.052423] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1134.059574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1135.072297] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1135.074702] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1135.081257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1136.099045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1141.219773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1141.222492] Lustre: Skipped 1 previous similar message [ 1146.335292] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1146.342988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1146.348735] Lustre: Skipped 1 previous similar message [ 1146.409660] Lustre: server umount lustre-OST0000 complete [ 1162.241586] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1162.253470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1162.257740] Lustre: Skipped 1 previous similar message [ 1168.512318] Lustre: server umount lustre-MDT0000 complete [ 1170.440349] LustreError: 54837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230914 with bad export cookie 17471696147796749214 [ 1170.443386] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1170.446825] LustreError: 54837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1176.641850] Lustre: server umount lustre-MDT0001 complete [ 1181.093455] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 1182.392508] Key type lgssc unregistered [ 1182.545679] LNet: 58220:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1182.550926] LNetError: 58220:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1182.557634] LNet: Removed LNI 192.168.206.146@tcp [ 1182.924659] Key type .llcrypt unregistered [ 1182.927203] Key type ._llcrypt unregistered [ 1193.113855] Key type ._llcrypt registered [ 1193.115665] Key type .llcrypt registered [ 1193.170873] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 1193.545844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1193.569428] alg: No test for adler32 (adler32-zlib) [ 1194.493082] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 1194.600382] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 1196.199168] Key type lgssc registered [ 1196.660463] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1199.825317] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1199.834502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1201.044044] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1202.614960] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1206.403586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1206.569870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1208.175793] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1210.586289] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1212.152714] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1215.087866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1215.255349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1217.674462] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1219.993756] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1226.542114] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1236.962210] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1236.962295] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1236.966096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1236.967430] LustreError: Skipped 1 previous similar message [ 1236.974817] Lustre: Skipped 1 previous similar message [ 1239.070912] Lustre: server umount lustre-OST0000 complete [ 1242.081150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1242.082117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1242.086843] Lustre: Skipped 1 previous similar message [ 1242.089343] Lustre: Skipped 1 previous similar message [ 1246.681516] Lustre: server umount lustre-MDT0000 complete [ 1247.200542] LustreError: 59605:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1247.208390] LustreError: 59605:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1248.222464] LustreError: 61218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230992 with bad export cookie 5958933808307844639 [ 1248.225893] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1248.226406] LustreError: 61218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1248.354768] Lustre: server umount lustre-MDT0001 complete [ 1252.576549] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 1253.809145] Key type lgssc unregistered [ 1253.934537] LNet: 62798:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1253.938387] LNetError: 62798:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1253.946428] LNet: Removed LNI 192.168.206.146@tcp [ 1254.267077] Key type .llcrypt unregistered [ 1254.268505] Key type ._llcrypt unregistered [ 1265.077904] Key type ._llcrypt registered [ 1265.079296] Key type .llcrypt registered [ 1265.122716] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 1265.490451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1265.524807] alg: No test for adler32 (adler32-zlib) [ 1266.413111] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 1266.525971] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 1268.119163] Key type lgssc registered [ 1268.575988] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1272.723650] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 22:23:35 (1781231015) [ 1275.584672] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1275.598655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1276.826967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1278.438826] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1282.129684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1282.305564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1283.861198] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1286.181546] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1287.743496] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1290.576229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1290.727538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1293.195932] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1295.347747] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1300.974856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1308.226375] Lustre: Failing over lustre-MDT0000 [ 1308.397902] Lustre: server umount lustre-MDT0000 complete [ 1311.052515] LustreError: 65622:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1311.057974] LustreError: 65622:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1311.202616] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1311.207892] Lustre: Skipped 1 previous similar message [ 1311.826904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1311.871809] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1311.931525] Lustre: *** cfs_fail_loc=135, val=0*** [ 1311.974264] LustreError: 67232:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 1311.978500] LustreError: 67232:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.206.146@tcp: cfg command failed: rc = -2 [ 1311.983612] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1311.983612] [ 1311.987152] LustreError: MGC192.168.206.146@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1311.993844] LustreError: 67220:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1311.997721] LustreError: 67220:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 1312.001874] LustreError: 67220:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 1312.004692] LustreError: 67220:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1312.016469] Lustre: server umount lustre-MDT0000 complete [ 1312.018498] LustreError: 67220:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1314.590284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1314.655976] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1314.755543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1314.781713] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1316.165512] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1316.216543] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1319.915117] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1319.927974] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1319.946392] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1323.487896] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1323.490819] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1323.496119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1325.025330] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1325.027929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1325.029372] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1327.068033] Lustre: server umount lustre-OST0000 complete [ 1330.144679] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1330.144968] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1330.154526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1330.157395] Lustre: Skipped 2 previous similar messages [ 1334.814903] Lustre: server umount lustre-MDT0000 complete [ 1335.265686] LustreError: 65175:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1335.274734] LustreError: 65175:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1336.374692] LustreError: 64443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231080 with bad export cookie 7100411910105966016 [ 1336.378943] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1336.379511] LustreError: 64443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1336.527378] Lustre: server umount lustre-MDT0001 complete [ 1341.061497] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 1342.330420] Key type lgssc unregistered [ 1342.477727] LNet: 68970:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1342.481955] LNetError: 68970:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1342.493931] LNet: Removed LNI 192.168.206.146@tcp [ 1342.910400] Key type .llcrypt unregistered [ 1342.912382] Key type ._llcrypt unregistered [ 1353.510363] Key type ._llcrypt registered [ 1353.512873] Key type .llcrypt registered [ 1353.585557] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 1353.909393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1353.965532] alg: No test for adler32 (adler32-zlib) [ 1354.845278] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 1354.941824] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 1356.535228] Key type lgssc registered [ 1357.002524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1360.314975] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1360.325721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1361.558368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1363.145387] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1366.944612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1367.123496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1368.790970] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1371.133743] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1372.707908] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1375.715159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1375.897084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1378.289371] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1380.949560] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1386.155433] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1388.719193] Lustre: Failing over lustre-MDT0000 [ 1388.885546] Lustre: server umount lustre-MDT0000 complete [ 1391.445382] LustreError: 70339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1391.455941] LustreError: 70339:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1391.593343] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1391.600824] Lustre: Skipped 2 previous similar messages [ 1393.666621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1393.744278] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1393.835137] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1393.913734] LustreError: 73086:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 1393.918990] LustreError: 73086:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.206.146@tcp: cfg command failed: rc = -9 [ 1393.925467] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1393.925467] [ 1393.935979] LustreError: MGC192.168.206.146@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1393.947516] LustreError: 73074:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1393.952740] LustreError: 73074:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -9 [ 1393.959843] LustreError: 73074:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 1393.964762] LustreError: 73074:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1393.985060] Lustre: server umount lustre-MDT0000 complete [ 1393.988129] LustreError: 73074:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1396.551145] LustreError: 70338:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1396.565253] LustreError: 70338:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1398.045579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1398.124927] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1398.256500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1398.295444] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1400.142915] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1401.670270] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1403.369811] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1403.380837] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 1403.402438] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1407.969285] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1407.975575] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1407.982828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1408.484982] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1408.489541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1408.491670] Lustre: Skipped 1 previous similar message [ 1410.780673] Lustre: server umount lustre-OST0000 complete [ 1413.601270] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1413.603487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1413.608398] Lustre: Skipped 2 previous similar messages [ 1413.608833] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1413.611692] Lustre: Skipped 1 previous similar message [ 1418.596218] Lustre: server umount lustre-MDT0000 complete [ 1418.720121] LustreError: 71063:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1418.727142] LustreError: 71063:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1420.113724] LustreError: 70324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231164 with bad export cookie 12713525649468222160 [ 1420.116482] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1420.120344] LustreError: 70324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1420.280978] Lustre: server umount lustre-MDT0001 complete [ 1424.692882] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 1425.921395] Key type lgssc unregistered [ 1426.065572] LNet: 74829:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1426.068713] LNetError: 74829:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1426.075579] LNet: Removed LNI 192.168.206.146@tcp [ 1426.426344] Key type .llcrypt unregistered [ 1426.428307] Key type ._llcrypt unregistered [ 1431.490835] Key type ._llcrypt registered [ 1431.492402] Key type .llcrypt registered [ 1431.536643] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 1431.657459] Key type .llcrypt unregistered [ 1431.659286] Key type ._llcrypt unregistered [ 1442.809508] Key type ._llcrypt registered [ 1442.810976] Key type .llcrypt registered [ 1442.867782] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 1443.459035] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1443.471490] alg: No test for adler32 (adler32-zlib) [ 1444.391109] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 1444.533495] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 1446.135156] Key type lgssc registered [ 1446.718322] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1450.346734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1450.357424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1451.612273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1453.501088] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1457.960772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1458.152215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1459.869606] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1462.253505] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1463.842741] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1467.075144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1467.245564] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1469.726091] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1472.148596] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1476.455655] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1483.745055] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1483.749275] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1483.757675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1486.817426] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1486.821395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1486.821929] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1488.221287] Lustre: server umount lustre-OST0000 complete [ 1491.936898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1491.937797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1491.944822] Lustre: Skipped 1 previous similar message [ 1491.947473] Lustre: Skipped 1 previous similar message [ 1497.056142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1497.058690] Lustre: Skipped 2 previous similar messages [ 1502.176048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1502.178916] Lustre: Skipped 1 previous similar message [ 1504.223270] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1504.286660] Lustre: server umount lustre-MDT0000 complete [ 1505.692911] LustreError: 76653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231249 with bad export cookie 3794557255572048765 [ 1505.696824] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1505.698238] LustreError: 76653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1505.820925] Lustre: server umount lustre-MDT0001 complete [ 1510.225127] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 1511.650285] Key type lgssc unregistered [ 1511.810573] LNet: 79844:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1511.814382] LNetError: 79844:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1511.822711] LNet: Removed LNI 192.168.206.146@tcp [ 1512.179242] Key type .llcrypt unregistered [ 1512.181120] Key type ._llcrypt unregistered [ 1522.840601] Key type ._llcrypt registered [ 1522.841724] Key type .llcrypt registered [ 1522.888668] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 1523.271920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1523.314555] alg: No test for adler32 (adler32-zlib) [ 1524.178808] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 1524.277807] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 1525.888135] Key type lgssc registered [ 1526.401066] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1529.839059] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1529.852827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1531.097707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1532.737836] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1536.239285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1536.421613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1538.113783] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1540.538632] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1542.151152] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1545.187593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1545.356757] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1547.754504] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1550.153716] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1556.649258] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 1562.080263] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1562.083931] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1562.094026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1562.099594] Lustre: Skipped 1 previous similar message [ 1565.335995] Lustre: server umount lustre-OST0000 complete [ 1567.200307] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1567.201302] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1567.204096] LustreError: Skipped 1 previous similar message [ 1567.204777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1567.211527] Lustre: Skipped 2 previous similar messages [ 1567.224290] Lustre: Skipped 1 previous similar message [ 1572.320649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1572.323488] Lustre: Skipped 1 previous similar message [ 1573.041369] Lustre: server umount lustre-MDT0000 complete [ 1574.812728] LustreError: 81196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231318 with bad export cookie 13886795473295501903 [ 1574.815901] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1574.818409] LustreError: 81196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1574.972472] Lustre: server umount lustre-MDT0001 complete [ 1579.914782] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 1581.303524] Key type lgssc unregistered [ 1581.446914] LNet: 84387:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1581.450499] LNetError: 84387:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1581.462467] LNet: Removed LNI 192.168.206.146@tcp [ 1581.848412] Key type .llcrypt unregistered [ 1581.850684] Key type ._llcrypt unregistered [ 1593.251097] Key type ._llcrypt registered [ 1593.252312] Key type .llcrypt registered [ 1593.310468] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 1593.700657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1593.736109] alg: No test for adler32 (adler32-zlib) [ 1594.625219] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 1594.749289] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 1596.359221] Key type lgssc registered [ 1596.932464] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1600.775498] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1600.786620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1601.995746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1603.545590] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1607.118308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1607.305850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1608.903329] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1611.195472] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1612.729425] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1615.631680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1615.806099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1618.326891] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1620.797232] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1627.046874] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 1637.346698] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1637.346888] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1637.349797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1637.352852] Lustre: Skipped 1 previous similar message [ 1641.179374] Lustre: server umount lustre-OST0000 complete [ 1643.488194] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1643.491946] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1643.498388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1643.500613] Lustre: Skipped 1 previous similar message [ 1647.584659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1647.584944] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1648.822378] Lustre: server umount lustre-MDT0000 complete [ 1650.462048] LustreError: 85742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231394 with bad export cookie 7609158835692257926 [ 1650.463396] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1650.466604] LustreError: 85742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1650.613789] Lustre: server umount lustre-MDT0001 complete [ 1654.936614] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 1656.202394] Key type lgssc unregistered [ 1656.349541] LNet: 88954:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1656.352943] LNetError: 88954:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1656.363072] LNet: Removed LNI 192.168.206.146@tcp [ 1656.725428] Key type .llcrypt unregistered [ 1656.726807] Key type ._llcrypt unregistered [ 1667.459294] Key type ._llcrypt registered [ 1667.460867] Key type .llcrypt registered [ 1667.516824] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 1668.169083] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1668.192338] alg: No test for adler32 (adler32-zlib) [ 1669.083557] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 1669.186424] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 1670.783196] Key type lgssc registered [ 1671.243305] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1674.301026] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1674.307816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1689.759481] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1694.879193] LustreError: 90301:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.146@tcp: failed processing log, type 4: rc = -110 [ 1721.503265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1723.807680] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1726.204913] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1729.029510] Lustre: server umount lustre-OST0000 complete [ 1732.602072] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 22:31:15 (1781231475) [ 1735.399251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1735.581258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1737.082953] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1740.504216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1740.665564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1742.162764] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1744.308236] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1745.790850] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1748.451667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1748.582623] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1750.526696] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1752.456085] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1760.876874] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 1771.490156] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1771.490171] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1771.496636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1771.500992] Lustre: Skipped 1 previous similar message [ 1773.027744] Lustre: server umount lustre-OST0000 complete [ 1776.610017] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1776.610286] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1776.617473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1776.621090] LustreError: Skipped 1 previous similar message [ 1776.625996] Lustre: Skipped 2 previous similar messages [ 1780.772784] Lustre: server umount lustre-MDT0000 complete [ 1781.729704] LustreError: 93969:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1781.735631] LustreError: 93969:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1782.311078] LustreError: 91803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231526 with bad export cookie 7208786097381519654 [ 1782.313267] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1782.316306] LustreError: 91803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1782.480233] Lustre: server umount lustre-MDT0001 complete [ 1786.659671] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 1788.020835] Key type lgssc unregistered [ 1788.174654] LNet: 95173:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1788.179464] LNetError: 95173:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1788.197389] LNet: Removed LNI 192.168.206.146@tcp [ 1788.583465] Key type .llcrypt unregistered [ 1788.584593] Key type ._llcrypt unregistered [ 1807.815874] Key type ._llcrypt registered [ 1807.817293] Key type .llcrypt registered [ 1807.869212] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 1808.216884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1808.227331] alg: No test for adler32 (adler32-zlib) [ 1809.111700] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 1809.209782] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 1810.807167] Key type lgssc registered [ 1811.315651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1815.190137] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1815.200162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1816.430848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1818.600820] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1823.695506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1823.917277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1826.187671] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1829.360200] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1831.379812] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1834.861076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1835.030612] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1837.911023] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1840.479863] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1841.192677] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 1856.480686] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1856.485347] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1856.493431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1858.534508] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1858.537355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1858.539422] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1863.650317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1863.653770] Lustre: Skipped 2 previous similar messages [ 1868.255404] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1868.341113] Lustre: server umount lustre-OST0000 complete [ 1870.304081] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1870.307628] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1870.315638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1873.889449] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1876.143354] Lustre: server umount lustre-MDT0000 complete [ 1877.769689] LustreError: 97299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231621 with bad export cookie 4990379671356047814 [ 1877.774366] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1877.774796] LustreError: 97299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1877.910189] Lustre: server umount lustre-MDT0001 complete [ 1882.973586] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 1884.463115] Key type lgssc unregistered [ 1884.593571] LNet: 100514:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1884.597318] LNetError: 100514:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1884.604633] LNet: Removed LNI 192.168.206.146@tcp [ 1885.017477] Key type .llcrypt unregistered [ 1885.020705] Key type ._llcrypt unregistered [ 1897.531984] Key type ._llcrypt registered [ 1897.534375] Key type .llcrypt registered [ 1897.601371] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 1898.434717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1898.493976] alg: No test for adler32 (adler32-zlib) [ 1899.441367] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 1899.605845] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 1901.239191] Key type lgssc registered [ 1902.224804] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1907.616769] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1907.627863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1923.103803] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1928.223202] LustreError: 101718:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.146@tcp: failed processing log, type 4: rc = -110 [ 1954.847181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1957.377602] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1960.039850] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1962.948146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1963.086235] LustreError: MGC192.168.206.146@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1963.092713] LustreError: 102629:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1963.096184] LustreError: 102629:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 1963.100984] LustreError: 102629:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 1963.141803] Lustre: server umount lustre-MDT0000 complete [ 1963.143414] LustreError: 102629:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1976.855609] Lustre: server umount lustre-OST0000 complete [ 1981.586886] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 1985.926573] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 2004.802334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2004.953881] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2004.981250] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2005.032260] Lustre: lustre-MDT0000: new disk, initializing [ 2005.087617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2005.095996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2007.211641] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2013.376365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2013.445385] Lustre: 105384:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2013.480990] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2013.483960] Lustre: Skipped 1 previous similar message [ 2013.525397] Lustre: lustre-MDT0001: new disk, initializing [ 2013.579113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2013.600914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2013.609978] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2016.135876] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2021.118428] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2023.243751] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2027.144411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2027.293657] Lustre: lustre-OST0000: new disk, initializing [ 2027.296239] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2027.300221] Lustre: 107125:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2027.350025] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2030.372531] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2033.175685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2033.182719] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2033.249548] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2034.910112] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2037.342935] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2037.461902] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2039.913409] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2040.033222] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2043.363886] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2043.364071] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2043.381567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2043.384625] Lustre: Skipped 1 previous similar message [ 2047.207449] Lustre: server umount lustre-OST0000 complete [ 2049.508123] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2049.514093] LustreError: Skipped 1 previous similar message [ 2049.517328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2049.525776] Lustre: Skipped 1 previous similar message [ 2049.529491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2053.601254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2053.603088] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2058.722862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2058.725952] Lustre: Skipped 2 previous similar messages [ 2063.327146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2063.441476] Lustre: server umount lustre-MDT0000 complete [ 2065.086869] LustreError: 106162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231809 with bad export cookie 461557841367805381 [ 2065.087937] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2065.091329] LustreError: 106162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2065.233638] Lustre: server umount lustre-MDT0001 complete [ 2068.922161] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 22:36:52 (1781231812) [ 2070.461737] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 2077.828814] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 2081.867849] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 2098.796751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2098.918526] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2098.935333] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2098.979602] Lustre: lustre-MDT0000: new disk, initializing [ 2099.026288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2099.036523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2101.069045] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2106.749596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2106.810602] Lustre: 111834:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2106.842269] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2106.844615] Lustre: Skipped 1 previous similar message [ 2106.886992] Lustre: lustre-MDT0001: new disk, initializing [ 2106.964922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2106.970083] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2108.852528] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2113.241198] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2115.036636] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2118.667261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2118.842306] Lustre: lustre-OST0000: new disk, initializing [ 2118.845539] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2118.854594] Lustre: 113573:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2118.917691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2118.920517] Lustre: Skipped 1 previous similar message [ 2120.406677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2120.416453] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2120.500055] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2121.913766] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2126.088062] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2128.267561] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2128.373287] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2130.204923] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2130.326431] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2136.034477] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2136.036120] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2136.037033] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2136.037985] LustreError: Skipped 1 previous similar message [ 2136.042630] Lustre: Skipped 1 previous similar message [ 2137.569892] Lustre: server umount lustre-OST0000 complete [ 2141.160862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2141.167083] Lustre: Skipped 1 previous similar message [ 2146.179411] Lustre: server umount lustre-MDT0000 complete [ 2146.275458] LustreError: 111847:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2146.284329] LustreError: 111847:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2148.559852] LustreError: 111826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231892 with bad export cookie 461557841367806249 [ 2148.564882] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2148.571708] LustreError: 111826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2148.844261] Lustre: server umount lustre-MDT0001 complete [ 2154.221525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2154.476498] LustreError: 115638:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2154.496617] LustreError: 115638:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2154.529426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2156.743707] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2159.587216] LustreError: 115639:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2161.258301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2163.743636] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2166.898640] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2169.170442] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2173.012623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2176.743991] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2180.142346] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2188.068788] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 2191.840788] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2191.844100] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2191.852565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2191.865948] Lustre: Skipped 1 previous similar message [ 2191.869976] Lustre: Skipped 3 previous similar messages [ 2192.438695] Lustre: server umount lustre-OST0000 complete [ 2209.247840] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2209.369293] Lustre: server umount lustre-MDT0000 complete [ 2211.459331] LustreError: 115619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231955 with bad export cookie 461557841367807131 [ 2211.466737] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2211.638719] Lustre: server umount lustre-MDT0001 complete [ 2216.641963] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 2218.046613] Key type lgssc unregistered [ 2218.231795] LNet: 118908:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2218.237486] LNetError: 118908:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2218.248064] LNet: Removed LNI 192.168.206.146@tcp [ 2218.680802] Key type .llcrypt unregistered [ 2218.682061] Key type ._llcrypt unregistered [ 2227.046435] Key type ._llcrypt registered [ 2227.049319] Key type .llcrypt registered [ 2227.095232] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 2234.929768] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 2235.451928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2235.464924] alg: No test for adler32 (adler32-zlib) [ 2236.384667] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 2236.519798] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 2238.159178] Key type lgssc registered [ 2238.867277] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2257.784311] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2257.799469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2258.938401] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2258.967794] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2259.017356] Lustre: lustre-MDT0000: new disk, initializing [ 2259.066056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2259.078711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2261.298940] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2267.180383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2267.250328] Lustre: 122003:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2267.300187] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2267.303340] Lustre: Skipped 1 previous similar message [ 2267.358691] Lustre: lustre-MDT0001: new disk, initializing [ 2267.416530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2267.443361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2267.449743] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2269.410068] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2274.043393] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2275.930883] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2279.060082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2279.199587] Lustre: lustre-OST0000: new disk, initializing [ 2279.203092] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2279.207295] Lustre: 123744:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2279.258840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2279.528400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2279.538364] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2279.595549] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2282.069401] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2286.333239] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2288.484130] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2288.597652] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2290.617808] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2290.751727] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2295.269671] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2295.274351] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2295.281269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2297.831709] Lustre: server umount lustre-OST0000 complete [ 2300.386505] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2300.393862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2300.396537] Lustre: Skipped 2 previous similar messages [ 2300.404982] Lustre: Skipped 2 previous similar messages [ 2305.504217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2305.512917] Lustre: Skipped 1 previous similar message [ 2310.624922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2310.629884] Lustre: Skipped 1 previous similar message [ 2314.207126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2314.300463] Lustre: server umount lustre-MDT0000 complete [ 2316.110911] LustreError: 121995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232060 with bad export cookie 3076340680067417175 [ 2316.112344] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2316.116808] LustreError: 121995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2316.303300] Lustre: server umount lustre-MDT0001 complete [ 2320.307436] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 22:41:03 (1781232063) [ 2323.540511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2323.769124] LustreError: 126007:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2323.780040] LustreError: 126007:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2323.808579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2325.843845] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2329.058099] LustreError: 126008:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2330.215456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2330.391693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2332.509060] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2335.502755] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2337.412288] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2338.784336] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2338.786605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2338.791419] Lustre: Skipped 1 previous similar message [ 2344.620876] Lustre: server umount lustre-MDT0000 complete [ 2345.953350] LustreError: 126007:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2345.959804] LustreError: 126007:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2346.342258] LustreError: 125989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232090 with bad export cookie 3076340680067418057 [ 2346.348885] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2346.356032] LustreError: 125989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2346.534455] Lustre: server umount lustre-MDT0001 complete [ 2350.750716] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 22:41:34 (1781232094) [ 2354.337938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2369.823549] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2374.943531] LustreError: 128190:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.146@tcp: failed processing log, type 4: rc = -110 [ 2400.607649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2403.477634] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2406.232452] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2407.274443] Lustre: server umount lustre-OST0000 complete [ 2411.402582] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 22:42:34 (1781232154) [ 2414.582305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2414.785557] LustreError: 129521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2416.670334] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2420.193052] LustreError: 129522:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2421.387224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2421.572962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2421.576112] Lustre: Skipped 1 previous similar message [ 2423.487991] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2426.360229] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2428.045514] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2431.339326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2434.504621] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2437.329775] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2447.328525] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2447.333717] LustreError: Skipped 1 previous similar message [ 2447.345585] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2447.357805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2447.361303] Lustre: Skipped 3 previous similar messages [ 2452.179451] Lustre: server umount lustre-MDT0000 complete [ 2454.553876] LustreError: 129501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232198 with bad export cookie 3076340680067418701 [ 2454.556694] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2454.561320] LustreError: 129501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2454.728234] Lustre: server umount lustre-MDT0001 complete [ 2467.280740] Lustre: server umount lustre-OST0000 complete [ 2472.246667] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 22:43:35 (1781232215) [ 2475.789151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2476.076130] LustreError: 132830:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2476.130825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2476.134789] Lustre: Skipped 1 previous similar message [ 2478.383364] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2483.142368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2485.493493] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2488.218969] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2490.309962] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2493.552028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2493.603437] LustreError: 134420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2493.612838] LustreError: 134420:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2496.490928] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2499.013796] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2500.951308] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2501.094985] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 2502.082322] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2504.071600] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2504.199167] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2506.211029] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2506.213612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2506.215101] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2506.221599] Lustre: Skipped 3 previous similar messages [ 2506.223710] Lustre: Skipped 3 previous similar messages [ 2507.113422] Lustre: server umount lustre-OST0000 complete [ 2509.280109] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2523.103125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2523.176596] Lustre: server umount lustre-MDT0000 complete [ 2524.718380] LustreError: 132812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232268 with bad export cookie 3076340680067419765 [ 2524.722170] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2524.722990] LustreError: 132812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2524.877954] Lustre: server umount lustre-MDT0001 complete [ 2528.972751] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 22:44:32 (1781232272) [ 2532.429573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2547.935499] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2553.055666] LustreError: 136616:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.146@tcp: failed processing log, type 4: rc = -110 [ 2578.655390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2578.660558] Lustre: Skipped 2 previous similar messages [ 2581.570806] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2584.348112] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2587.821085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2588.050129] LustreError: 136641:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2588.141262] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 2589.878986] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2593.849956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2595.994319] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2598.978071] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2601.088815] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2603.249541] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2603.382626] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2605.574620] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2605.707280] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2609.633272] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2609.633735] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2609.635211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2609.635215] Lustre: Skipped 7 previous similar messages [ 2609.649383] Lustre: Skipped 3 previous similar messages [ 2612.840227] Lustre: server umount lustre-MDT0000 complete [ 2614.520352] LustreError: 136622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232358 with bad export cookie 3076340680067420787 [ 2614.523901] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2614.524964] LustreError: 136622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2614.688973] Lustre: server umount lustre-MDT0001 complete [ 2626.788814] Lustre: server umount lustre-OST0000 complete [ 2630.386243] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 22:46:13 (1781232373) [ 2633.378588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2648.863603] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2653.984489] LustreError: 140432:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.146@tcp: failed processing log, type 4: rc = -110 [ 2681.938850] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2684.213227] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2687.308327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2687.530756] LustreError: 140457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2687.538269] LustreError: 140457:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2687.620621] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 2689.301939] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2693.100501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2694.894557] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2697.224694] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2698.782275] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2701.780658] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2701.874409] Lustre: lustre-OST0001: new disk, initializing [ 2701.877051] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2701.882175] Lustre: 142961:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2704.533200] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2707.457652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2707.463776] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2707.520517] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2708.504179] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2710.501661] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2710.603115] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2712.716337] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2712.858534] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2714.080657] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2714.085141] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2714.090487] Lustre: Skipped 1 previous similar message [ 2714.094207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2714.098727] Lustre: Skipped 2 previous similar messages [ 2719.876492] Lustre: server umount lustre-OST0000 complete [ 2727.657867] Lustre: server umount lustre-OST0001 complete [ 2735.548667] Lustre: server umount lustre-MDT0000 complete [ 2737.326808] LustreError: 142960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232481 with bad export cookie 3076340680067421739 [ 2737.330310] LustreError: 142960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2737.330785] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2737.477736] Lustre: server umount lustre-MDT0001 complete [ 2748.786802] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 22:48:12 (1781232492) [ 2749.570072] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 2750.299163] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 2750.990966] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 22:48:14 (1781232494) [ 2754.241209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2754.408119] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2754.445549] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2754.564844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2754.568238] Lustre: Skipped 6 previous similar messages [ 2756.546568] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2760.952166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2761.016951] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2761.033307] Lustre: 146367:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2763.173859] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2765.882612] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2767.619869] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2770.502692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2770.601164] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2773.204234] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2775.671210] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2777.410658] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2778.144781] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 2778.531367] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2780.138119] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2780.230247] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2783.202015] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2783.203601] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2783.208020] LustreError: Skipped 4 previous similar messages [ 2783.214534] Lustre: Skipped 5 previous similar messages [ 2787.104607] Lustre: server umount lustre-OST0000 complete [ 2802.268934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2803.429157] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2803.434668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2803.435303] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2803.446693] Lustre: Skipped 1 previous similar message [ 2804.760433] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2806.956899] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2808.943669] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2809.046582] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2810.737596] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2810.839543] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2812.575545] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff994343440800.ost_server_uuid 50 [ 2813.256777] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff994343440800.ost_server_uuid in FULL state after 0 sec [ 2813.322412] LustreError: 146372:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.206.46@tcp) returned error from blocking AST (req@000000007dcac3c1 x1867757104858624 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000007c31c05a/0x2ab15b8ca3809f77 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.206.46@tcp remote: 0xc0da1917f0c57b06 expref: 5 pid: 146371 timeout: 2913 lvb_type: 0 lru_score: 0 lru_type: 0 [ 2813.338354] LustreError: lustre-MDT0000: A client on nid 192.168.206.46@tcp was evicted due to a lock blocking callback time out: rc -107 [ 2813.342687] LustreError: 146362:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.206.46@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000007c31c05a/0x2ab15b8ca3809f77 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.206.46@tcp remote: 0xc0da1917f0c57b06 expref: 6 pid: 146371 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 2819.042063] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2819.050539] LustreError: Skipped 1 previous similar message [ 2820.818968] Lustre: server umount lustre-OST0000 complete [ 2828.724179] Lustre: server umount lustre-MDT0000 complete [ 2829.280338] LustreError: 146377:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2829.288207] LustreError: 146377:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2830.259930] LustreError: 146358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232574 with bad export cookie 3076340680067422922 [ 2830.263059] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2830.265142] LustreError: 146358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2836.498725] Lustre: server umount lustre-MDT0001 complete [ 2841.210291] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 2842.445607] Key type lgssc unregistered [ 2842.588518] LNet: 151843:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2842.592469] LNetError: 151843:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2842.600587] LNet: Removed LNI 192.168.206.146@tcp [ 2842.961478] Key type .llcrypt unregistered [ 2842.963033] Key type ._llcrypt unregistered [ 2853.203877] Key type ._llcrypt registered [ 2853.205139] Key type .llcrypt registered [ 2853.264316] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 2853.619989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2853.663664] alg: No test for adler32 (adler32-zlib) [ 2854.560466] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 2854.677411] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 2856.287151] Key type lgssc registered [ 2856.775542] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2859.908683] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2859.925936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2861.177525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2862.649673] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2866.286620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2866.472335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2868.298304] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2870.751229] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2872.280812] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2875.087557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2875.265087] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2877.532566] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2879.849792] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2886.570898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 2893.000789] Lustre: Failing over lustre-MDT0000 [ 2893.115727] Lustre: server umount lustre-MDT0000 complete [ 2896.865876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2896.868510] LustreError: 153241:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2896.871597] Lustre: Skipped 2 previous similar messages [ 2896.880302] LustreError: 153241:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2901.985362] LustreError: 153217:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2901.994140] LustreError: 153217:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2904.275683] LustreError: 153198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.46@tcp arrived at 1781232648 with bad export cookie 12511426125249198157 [ 2907.106590] LustreError: 153929:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2907.113340] LustreError: 153929:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2907.483372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2907.549436] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2907.686692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2907.721417] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2909.513865] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2910.486295] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2910.492108] Lustre: lustre-MDT0000: Denying connection for new client 40dd2ff9-804b-4258-9a55-68803d6db720 (at 192.168.206.46@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2912.744406] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2917.857312] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2917.863816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2922.464135] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2922.469791] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2922.475437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2923.551841] Lustre: server umount lustre-OST0000 complete [ 2925.078027] LustreError: 156604:0:(ldlm_lib.c:2985:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2925.081420] Lustre: 155971:0:(ldlm_lib.c:2388:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2925.085266] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2925.092278] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2925.097271] LustreError: 155971:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000da1e6987 x1867757752716288/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2925.105482] LustreError: 155971:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 2925.109783] LustreError: 155971:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 2925.114387] Lustre: lustre-MDT0000: Recovery over after 0:15, of 2 clients 0 recovered and 2 were evicted. [ 2925.226198] Lustre: server umount lustre-MDT0000 complete [ 2926.753948] LustreError: 153196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232670 with bad export cookie 12511426125249198486 [ 2926.757552] LustreError: 153929:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2926.758603] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2926.761061] LustreError: 153196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2932.930802] Lustre: server umount lustre-MDT0001 complete [ 2937.119842] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 2938.354504] Key type lgssc unregistered [ 2938.507930] LNet: 157340:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2938.512248] LNetError: 157340:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2938.525085] LNet: Removed LNI 192.168.206.146@tcp [ 2938.851655] Key type .llcrypt unregistered [ 2938.853445] Key type ._llcrypt unregistered [ 2949.441084] Key type ._llcrypt registered [ 2949.442717] Key type .llcrypt registered [ 2949.491367] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 2949.887030] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2949.922940] alg: No test for adler32 (adler32-zlib) [ 2950.804365] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 2950.911784] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 2952.511215] Key type lgssc registered [ 2953.019231] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2956.604135] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2956.614976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2957.826494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2959.472519] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2963.653251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2963.833822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2965.453275] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2967.582464] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2969.147279] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2971.942774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2972.098350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2974.431933] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2976.722282] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2982.375270] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 2987.489455] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2987.491284] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2987.498906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2990.427564] Lustre: server umount lustre-OST0000 complete [ 2992.612077] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2992.613188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2992.622370] Lustre: Skipped 2 previous similar messages [ 2992.624552] Lustre: Skipped 1 previous similar message [ 2997.729137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2997.733110] Lustre: Skipped 2 previous similar messages [ 2998.189209] Lustre: server umount lustre-MDT0000 complete [ 2999.858715] LustreError: 160862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232743 with bad export cookie 4712765209137333481 [ 2999.863550] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2999.865080] LustreError: 160862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3000.004210] Lustre: server umount lustre-MDT0001 complete [ 3004.316291] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 3005.686774] Key type lgssc unregistered [ 3005.844515] LNet: 161883:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3005.848598] LNetError: 161883:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3005.857756] LNet: Removed LNI 192.168.206.146@tcp [ 3006.236739] Key type .llcrypt unregistered [ 3006.238573] Key type ._llcrypt unregistered [ 3021.248128] Key type ._llcrypt registered [ 3021.251410] Key type .llcrypt registered [ 3021.301601] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 3021.848831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3021.863185] alg: No test for adler32 (adler32-zlib) [ 3022.768950] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 3022.880314] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 3024.481128] Key type lgssc registered [ 3025.011916] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3028.693281] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3028.706889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3029.952715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3032.057363] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3036.456579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3036.658329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3038.860563] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3041.366407] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3042.856281] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3045.675256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3045.837491] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3048.148914] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3050.415630] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3055.077524] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 3060.193768] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3060.194511] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3060.200244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3060.205202] Lustre: Skipped 1 previous similar message [ 3060.210146] Lustre: Skipped 1 previous similar message [ 3063.526720] Lustre: server umount lustre-OST0000 complete [ 3065.312523] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3065.318767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3065.321105] Lustre: Skipped 1 previous similar message [ 3070.434057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3070.437519] Lustre: Skipped 1 previous similar message [ 3071.415725] Lustre: server umount lustre-MDT0000 complete [ 3072.996417] LustreError: 163525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232817 with bad export cookie 10513304173365265984 [ 3072.999755] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3073.001706] LustreError: 163525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3073.122391] Lustre: server umount lustre-MDT0001 complete [ 3077.487674] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 3078.821286] Key type lgssc unregistered [ 3078.988117] LNet: 166716:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3078.992752] LNetError: 166716:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3079.002537] LNet: Removed LNI 192.168.206.146@tcp [ 3079.394127] Key type .llcrypt unregistered [ 3079.396847] Key type ._llcrypt unregistered [ 3089.612499] Key type ._llcrypt registered [ 3089.614125] Key type .llcrypt registered [ 3089.663482] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 3090.169600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3090.185001] alg: No test for adler32 (adler32-zlib) [ 3091.063539] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 3091.178610] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 3092.783665] Key type lgssc registered [ 3093.267723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3097.852421] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3097.863413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3099.046668] Lustre: *** cfs_fail_loc=135, val=0*** [ 3099.089427] LustreError: 168200:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 3099.092917] LustreError: 168200:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.206.146@tcp: cfg command failed: rc = -2 [ 3099.097720] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 3099.097720] [ 3099.101389] LustreError: MGC192.168.206.146@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3099.108511] LustreError: 168161:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3099.112514] LustreError: 168161:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 3099.116099] LustreError: 168161:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 3099.119256] LustreError: 168161:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 3099.165411] Lustre: server umount lustre-MDT0000 complete [ 3099.167663] LustreError: 168161:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3102.101105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3102.332813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3103.946399] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3107.803387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3108.001832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3109.814890] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3112.168436] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3113.959771] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3117.537433] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3117.539335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3117.545301] Lustre: Skipped 1 previous similar message [ 3118.562208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3118.565569] Lustre: Skipped 1 previous similar message [ 3123.680922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3123.684185] Lustre: Skipped 1 previous similar message [ 3128.800543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3128.803585] Lustre: Skipped 1 previous similar message [ 3129.311136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3129.390602] Lustre: server umount lustre-MDT0000 complete [ 3130.939286] LustreError: 168553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232875 with bad export cookie 12340155607987076303 [ 3130.942355] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3130.945761] LustreError: 168553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3131.082268] Lustre: server umount lustre-MDT0001 complete [ 3135.339416] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 3136.605398] Key type lgssc unregistered [ 3136.750834] LNet: 170648:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3136.754889] LNetError: 170648:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3136.761517] LNet: Removed LNI 192.168.206.146@tcp [ 3137.088978] Key type .llcrypt unregistered [ 3137.090171] Key type ._llcrypt unregistered [ 3146.177758] Key type ._llcrypt registered [ 3146.179617] Key type .llcrypt registered [ 3146.232388] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 3146.377292] Key type .llcrypt unregistered [ 3146.381927] Key type ._llcrypt unregistered [ 3154.483392] Key type ._llcrypt registered [ 3154.484809] Key type .llcrypt registered [ 3154.535747] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 3154.987321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3154.995913] alg: No test for adler32 (adler32-zlib) [ 3155.895722] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 3156.005863] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 3157.599167] Key type lgssc registered [ 3158.112232] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3161.613688] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3161.623292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3177.119608] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3182.239204] LustreError: 172722:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.146@tcp: failed processing log, type 4: rc = -110 [ 3208.863170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3211.071460] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3213.686587] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3216.677791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3216.956093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3216.991235] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 3218.784605] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3221.989406] LustreError: 172748:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3221.997201] LustreError: 172748:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3222.746466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3222.968697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3224.782671] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3227.144641] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3228.798810] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3230.047538] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3237.344412] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3237.349043] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3237.357690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3238.368237] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3238.371656] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3238.378784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3239.393547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3239.395844] Lustre: Skipped 1 previous similar message [ 3244.513893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3244.518969] Lustre: Skipped 1 previous similar message [ 3249.638068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3249.642742] Lustre: Skipped 1 previous similar message [ 3251.167104] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3251.224731] Lustre: server umount lustre-OST0000 complete [ 3253.728311] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3253.731536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3259.872078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3259.875024] Lustre: Skipped 4 previous similar messages [ 3267.039130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3267.106151] Lustre: server umount lustre-MDT0000 complete [ 3268.755255] LustreError: 172728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233012 with bad export cookie 119428627041828488 [ 3268.759023] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3268.761735] LustreError: 172728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3268.902100] Lustre: server umount lustre-MDT0001 complete [ 3273.261116] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 3274.586184] Key type lgssc unregistered [ 3274.744175] LNet: 176332:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3274.752250] LNetError: 176332:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3274.764438] LNet: Removed LNI 192.168.206.146@tcp [ 3275.165973] Key type .llcrypt unregistered [ 3275.167750] Key type ._llcrypt unregistered [ 3285.804772] Key type ._llcrypt registered [ 3285.806025] Key type .llcrypt registered [ 3285.860335] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 3286.345060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3286.356906] alg: No test for adler32 (adler32-zlib) [ 3287.238468] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 3287.346794] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 3288.943179] Key type lgssc registered [ 3289.401990] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3292.796343] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3292.804775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3294.056344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3295.790025] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3299.668968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3299.855412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3301.617933] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3303.911628] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3305.626077] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3308.512737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3308.704717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3311.056943] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3313.426402] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3318.951211] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 3321.704864] Lustre: Failing over lustre-MDT0000 [ 3321.873195] Lustre: server umount lustre-MDT0000 complete [ 3324.387724] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3324.389379] LustreError: 178854:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3324.392833] Lustre: Skipped 2 previous similar messages [ 3324.400629] LustreError: 178854:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3325.263189] LustreError: 179316:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3329.506590] LustreError: 177700:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3329.516254] LustreError: 177700:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3334.624246] LustreError: 177703:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3334.635240] LustreError: 177703:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3336.411701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3336.493236] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3336.638942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3336.682113] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3338.671630] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3340.620956] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3341.117740] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 3341.796877] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3341.806667] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3341.824960] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 3348.195523] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 3356.129030] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3356.134403] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3356.143504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3357.152738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3357.155935] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3357.157269] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3362.174312] Lustre: server umount lustre-OST0000 complete [ 3367.394162] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3367.400451] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3367.400467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3367.400521] Lustre: Skipped 1 previous similar message [ 3367.401178] Lustre: Skipped 1 previous similar message [ 3370.396799] Lustre: server umount lustre-MDT0000 complete [ 3372.517084] LustreError: 177704:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3372.525393] LustreError: 177704:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3372.541448] LustreError: 181218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233116 with bad export cookie 15503419367887957846 [ 3372.546731] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3372.548326] LustreError: 181218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3372.743797] Lustre: server umount lustre-MDT0001 complete [ 3379.245371] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 3381.318965] Key type lgssc unregistered [ 3381.534791] LNet: 182258:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3381.540068] LNetError: 182258:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3381.554225] LNet: Removed LNI 192.168.206.146@tcp [ 3382.096320] Key type .llcrypt unregistered [ 3382.098449] Key type ._llcrypt unregistered [ 3395.272461] Key type ._llcrypt registered [ 3395.274237] Key type .llcrypt registered [ 3395.323888] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 3395.713592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3395.781255] alg: No test for adler32 (adler32-zlib) [ 3396.712049] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 3396.858859] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 3398.487374] Key type lgssc registered [ 3399.054454] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3403.068594] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3403.083035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3404.368460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3406.105230] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3410.156734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3410.337989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3412.014372] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3414.521424] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3416.106903] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3419.305665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3419.500021] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3422.053780] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3424.812931] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3429.738636] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 3430.778261] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3434.448645] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 3443.820278] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3450.338889] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3450.339378] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3450.345197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3450.356298] Lustre: Skipped 1 previous similar message [ 3455.457726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3455.463502] Lustre: Skipped 1 previous similar message [ 3455.526791] Lustre: server umount lustre-OST0000 complete [ 3460.577724] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3460.579687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3460.584420] Lustre: Skipped 2 previous similar messages [ 3463.350792] Lustre: server umount lustre-MDT0000 complete [ 3465.199682] LustreError: 183613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233209 with bad export cookie 12270643059060027500 [ 3465.200660] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3465.208232] LustreError: 183613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3465.354370] Lustre: server umount lustre-MDT0001 complete [ 3470.350512] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 3471.806535] Key type lgssc unregistered [ 3471.980725] LNet: 187114:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3471.984765] LNetError: 187114:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3471.999437] LNet: Removed LNI 192.168.206.146@tcp [ 3472.469990] Key type .llcrypt unregistered [ 3472.473377] Key type ._llcrypt unregistered [ 3482.324349] Key type ._llcrypt registered [ 3482.325867] Key type .llcrypt registered [ 3482.375594] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 3482.880912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3482.906944] alg: No test for adler32 (adler32-zlib) [ 3483.858634] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 3484.000714] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 3485.623177] Key type lgssc registered [ 3486.158632] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3489.852275] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3489.868188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3491.133347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3493.124102] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3497.146891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3497.382753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3499.379878] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3502.758518] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3504.915135] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3509.132402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3509.415235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3512.582986] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3515.379585] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3516.590500] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 3519.741688] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 3527.239387] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3533.702823] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 3542.558901] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 3548.641802] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3548.647882] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3548.649496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3548.655100] Lustre: Skipped 1 previous similar message [ 3553.766031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3553.769727] Lustre: Skipped 1 previous similar message [ 3554.732894] Lustre: server umount lustre-OST0000 complete [ 3558.880935] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3558.881171] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3558.882097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3558.895288] Lustre: Skipped 2 previous similar messages [ 3562.785819] Lustre: server umount lustre-MDT0000 complete [ 3564.000535] LustreError: 189447:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3564.008989] LustreError: 189447:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3564.418505] LustreError: 191924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233308 with bad export cookie 12863735538460599011 [ 3564.420662] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3564.423756] LustreError: 191924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3564.569905] Lustre: server umount lustre-MDT0001 complete [ 3569.588909] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 3571.059654] Key type lgssc unregistered [ 3571.241776] LNet: 193156:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3571.247776] LNetError: 193156:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3571.263552] LNet: Removed LNI 192.168.206.146@tcp [ 3571.746526] Key type .llcrypt unregistered [ 3571.748805] Key type ._llcrypt unregistered [ 3584.082659] Key type ._llcrypt registered [ 3584.084672] Key type .llcrypt registered [ 3584.139467] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 3584.590096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3584.662385] alg: No test for adler32 (adler32-zlib) [ 3585.576330] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 3585.696041] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 3587.335163] Key type lgssc registered [ 3587.911867] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3591.725145] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3591.735584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3592.954889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3594.650732] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3598.733962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3598.953109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3601.014958] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3603.977544] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3605.856165] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3609.408358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3609.604735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3612.592663] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3615.306385] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3618.859076] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 3628.012113] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3628.025469] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3628.033101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3628.036940] Lustre: Skipped 1 previous similar message [ 3632.044077] Lustre: server umount lustre-OST0000 complete [ 3634.655958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3634.661292] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3634.669864] Lustre: Skipped 1 previous similar message [ 3634.673445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3638.246646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3638.248598] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3640.611236] Lustre: server umount lustre-MDT0000 complete [ 3642.687063] LustreError: 194511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233386 with bad export cookie 10573543222130957665 [ 3642.688827] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3642.692752] LustreError: 194511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3642.897648] Lustre: server umount lustre-MDT0001 complete [ 3648.518904] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 3649.933412] Key type lgssc unregistered [ 3650.089955] LNet: 197853:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3650.093649] LNetError: 197853:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3650.105613] LNet: Removed LNI 192.168.206.146@tcp [ 3650.469510] Key type .llcrypt unregistered [ 3650.470941] Key type ._llcrypt unregistered [ 3660.697516] Key type ._llcrypt registered [ 3660.699413] Key type .llcrypt registered [ 3660.764296] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 3661.269159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3661.305205] alg: No test for adler32 (adler32-zlib) [ 3662.229372] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 3662.366609] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 3663.969085] Key type lgssc registered [ 3664.565119] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3668.438105] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3668.447491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3669.718136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3671.517881] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3675.803367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3676.010303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3677.980240] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3681.019725] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3683.272631] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3687.029449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3687.239742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3690.435199] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3693.433333] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3694.445247] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 3706.849996] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3706.852223] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3706.854355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3706.860017] Lustre: Skipped 1 previous similar message [ 3707.568332] Lustre: server umount lustre-OST0000 complete [ 3711.488916] Lustre: server umount lustre-MDT0000 complete [ 3711.968105] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3711.969924] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3711.972264] LustreError: 200391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3711.978213] Lustre: Skipped 1 previous similar message [ 3711.991430] LustreError: 200391:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3713.422885] LustreError: 199058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233457 with bad export cookie 2933674342000977754 [ 3713.424357] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3713.428901] LustreError: 199058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3715.595389] Lustre: server umount lustre-MDT0001 complete [ 3720.785877] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 3722.229683] Key type lgssc unregistered [ 3722.396957] LNet: 202397:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3722.402680] LNetError: 202397:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3722.413234] LNet: Removed LNI 192.168.206.146@tcp [ 3722.857388] Key type .llcrypt unregistered [ 3722.859379] Key type ._llcrypt unregistered [ 3732.775509] Key type ._llcrypt registered [ 3732.776870] Key type .llcrypt registered [ 3732.825272] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 3733.377378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3733.389811] alg: No test for adler32 (adler32-zlib) [ 3734.311197] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 3734.424141] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 3736.047173] Key type lgssc registered [ 3736.639610] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3740.801914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3740.817119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3742.143716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3744.250111] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3748.469105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3748.673143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3750.952473] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3754.004812] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3756.093907] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3759.524512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3759.733166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3762.488628] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3765.209663] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3767.915258] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 3776.584841] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3776.636518] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3776.760558] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3780.040735] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3782.831230] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3783.601941] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3794.345275] Lustre: Permanently deactivating lustre-OST0001 [ 3794.353822] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3803.667473] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3809.969838] Lustre: Permanently reactivating lustre-OST0001 [ 3809.976954] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 3809.980598] Lustre: Skipped 2 previous similar messages [ 3812.221653] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3812.228153] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3812.232585] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3812.238643] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3812.255233] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3818.980377] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3818.982677] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3818.986742] Lustre: Skipped 2 previous similar messages [ 3818.992306] Lustre: Skipped 1 previous similar message [ 3824.102239] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3824.105060] Lustre: Skipped 1 previous similar message [ 3824.840558] Lustre: server umount lustre-OST0001 complete [ 3829.218235] LustreError: 207101:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3829.218683] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3829.219877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3829.228566] LustreError: 207101:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3829.233096] Lustre: Skipped 1 previous similar message [ 3832.625966] Lustre: server umount lustre-OST0000 complete [ 3834.337059] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3834.337919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3834.341943] Lustre: Skipped 1 previous similar message [ 3834.347377] Lustre: Skipped 2 previous similar messages [ 3839.456097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3839.459091] Lustre: Skipped 1 previous similar message [ 3840.431593] Lustre: server umount lustre-MDT0000 complete [ 3842.312757] LustreError: 205242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233586 with bad export cookie 11299589841058415666 [ 3842.316144] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3842.320533] LustreError: 205242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3842.488452] Lustre: server umount lustre-MDT0001 complete [ 3847.480567] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 3848.905349] Key type lgssc unregistered [ 3849.107689] LNet: 208237:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3849.112650] LNetError: 208237:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3849.123341] LNet: Removed LNI 192.168.206.146@tcp [ 3849.569337] Key type .llcrypt unregistered [ 3849.571146] Key type ._llcrypt unregistered [ 3870.345239] Key type ._llcrypt registered [ 3870.346366] Key type .llcrypt registered [ 3870.397429] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 3870.802281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3870.844330] alg: No test for adler32 (adler32-zlib) [ 3871.745549] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 3871.850194] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 3873.439185] Key type lgssc registered [ 3873.983486] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3878.171623] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3878.183410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3879.341357] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3879.355099] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3879.440672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3881.573898] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3885.712392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3885.776392] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3885.793551] Lustre: 210399:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3885.939185] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3887.664919] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3890.321372] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3891.895596] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3894.951397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3895.052837] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3895.164183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3897.677209] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3900.015165] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3900.454803] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 3902.819830] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 3908.530272] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 3918.398446] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 3928.321196] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3937.193729] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3959.566824] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 3959.573095] Lustre: Skipped 2 previous similar messages [ 3992.741664] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3992.747612] Lustre: Skipped 3 previous similar messages [ 4061.718596] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 4061.726666] Lustre: Skipped 8 previous similar messages [ 4074.465279] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4074.467996] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4074.468945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4074.481357] Lustre: Skipped 1 previous similar message [ 4079.590702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4079.593321] Lustre: Skipped 2 previous similar messages [ 4080.317613] Lustre: server umount lustre-OST0000 complete [ 4084.705042] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4084.710383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4084.717339] Lustre: Skipped 1 previous similar message [ 4088.380913] Lustre: server umount lustre-MDT0000 complete [ 4090.258950] LustreError: 213303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233834 with bad export cookie 14844921751553035828 [ 4090.262344] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4090.268563] LustreError: 213303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4090.429896] Lustre: server umount lustre-MDT0001 complete [ 4095.769132] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 4097.303361] Key type lgssc unregistered [ 4097.470436] LNet: 214848:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4097.474451] LNetError: 214848:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4098.532643] LNet: Removed LNI 192.168.206.146@tcp [ 4098.946050] Key type .llcrypt unregistered [ 4098.947544] Key type ._llcrypt unregistered [ 4111.368620] Key type ._llcrypt registered [ 4111.371793] Key type .llcrypt registered [ 4111.452200] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 4112.054774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4112.090343] alg: No test for adler32 (adler32-zlib) [ 4113.013804] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 4113.179091] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 4114.823271] Key type lgssc registered [ 4115.457439] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4119.665361] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4119.673471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4120.912100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4122.957052] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 4127.282855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4127.512406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4129.576443] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 4132.440777] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4134.369897] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4137.920913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4138.110662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4140.850772] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 4143.635689] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4145.320510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 4163.043731] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4163.043850] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4163.055114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4163.062510] Lustre: Skipped 1 previous similar message [ 4164.131687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4164.134517] Lustre: Skipped 1 previous similar message [ 4167.719887] Lustre: server umount lustre-OST0000 complete [ 4170.656781] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4170.668390] Lustre: Skipped 1 previous similar message [ 4170.671821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4170.674570] Lustre: Skipped 1 previous similar message [ 4175.784898] Lustre: server umount lustre-MDT0000 complete [ 4175.840190] LustreError: 216216:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4175.849485] LustreError: 216216:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4177.588225] LustreError: 216202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233921 with bad export cookie 14279718854729773051 [ 4177.589059] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4177.597292] LustreError: 216202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4177.784724] Lustre: server umount lustre-MDT0001 complete [ 4182.690538] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 4183.998624] Key type lgssc unregistered [ 4184.162662] LNet: 219559:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4184.166611] LNetError: 219559:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4184.176476] LNet: Removed LNI 192.168.206.146@tcp [ 4184.566933] Key type .llcrypt unregistered [ 4184.569543] Key type ._llcrypt unregistered [ 4195.816926] Key type ._llcrypt registered [ 4195.818779] Key type .llcrypt registered [ 4195.878783] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 4196.345889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4196.399888] alg: No test for adler32 (adler32-zlib) [ 4197.294649] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 4197.407754] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 4199.015178] Key type lgssc registered [ 4199.595580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4466.138091] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 4467.734489] Key type lgssc unregistered [ 4467.895544] LNet: 221524:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4467.901341] LNetError: 221524:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4467.914984] LNet: Removed LNI 192.168.206.146@tcp [ 4468.366913] Key type .llcrypt unregistered [ 4468.369286] Key type ._llcrypt unregistered [ 4480.732354] Key type ._llcrypt registered [ 4480.733834] Key type .llcrypt registered [ 4480.786468] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 4489.431520] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 4489.966542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4489.974459] alg: No test for adler32 (adler32-zlib) [ 4490.878608] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 4490.995308] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 4492.599364] Key type lgssc registered [ 4493.265931] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4508.320965] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 4380 sec ========== 23:17:31 (1781234251) [ 4509.116899] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:17:32 (1781234252) === [ 4510.549812] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:17:33 (1781234253) === [ 4528.582718] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 4529.965744] Key type lgssc unregistered [ 4530.133151] LNet: 226681:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4530.137220] LNetError: 226681:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4530.147624] LNet: Removed LNI 192.168.206.146@tcp [ 4530.475469] Key type .llcrypt unregistered [ 4530.477601] Key type ._llcrypt unregistered