[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 506925107 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.003267] x2apic enabled [ 0.004005] Switched APIC routing to physical x2apic. [ 0.005009] kvm-guest: setup PV IPIs [ 0.008646] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010005] pid_max: default: 32768 minimum: 301 [ 0.011106] LSM: Security Framework initializing [ 0.012093] Yama: becoming mindful. [ 0.013024] SELinux: Initializing. [ 0.014092] *** VALIDATE selinux *** [ 0.023478] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.030251] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.032069] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033080] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034082] *** VALIDATE tmpfs *** [ 0.035384] *** VALIDATE proc *** [ 0.036188] *** VALIDATE cgroup *** [ 0.037008] *** VALIDATE cgroup2 *** [ 0.038082] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039000] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040024] Spectre V2 : User space: Vulnerable [ 0.041004] Speculative Store Bypass: Vulnerable [ 0.043144] debug: unmapping init [mem 0xffffffff84a59000-0xffffffff84a60fff] [ 0.045156] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046497] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047016] ... version: 2 [ 0.047957] ... bit width: 48 [ 0.048008] ... generic registers: 4 [ 0.049000] ... value mask: 0000ffffffffffff [ 0.050008] ... max period: 00007fffffffffff [ 0.051007] ... fixed-purpose events: 3 [ 0.051879] ... event mask: 000000070000000f [ 0.052276] rcu: Hierarchical SRCU implementation. [ 0.054042] smp: Bringing up secondary CPUs ... [ 0.055359] x86: Booting SMP configuration: [ 0.056012] .... node #0, CPUs: #1 #2 #3 [ 0.061078] smp: Brought up 1 node, 4 CPUs [ 0.063030] smpboot: Max logical packages: 1 [ 0.064008] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.139017] node 0 deferred pages initialised in 73ms [ 0.140714] devtmpfs: initialized [ 0.142221] x86/mm: Memory block size: 128MB [ 0.145848] gcov: version magic: 0x41383552 [ 0.149073] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.152060] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.155230] pinctrl core: initialized pinctrl subsystem [ 0.157165] [ 0.157784] ************************************************************* [ 0.161007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163006] ** ** [ 0.166007] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.168006] ** ** [ 0.171007] ** This means that this kernel is built to expose internal ** [ 0.173008] ** IOMMU data structures, which may compromise security on ** [ 0.177010] ** your system. ** [ 0.179006] ** ** [ 0.182007] ** If you see this message and you are not debugging the ** [ 0.184007] ** kernel, report this immediately to your vendor! ** [ 0.187007] ** ** [ 0.189006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.192007] ************************************************************* [ 0.194628] NET: Registered protocol family 16 [ 0.196491] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.199042] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.202100] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.204011] cpuidle: using governor menu [ 0.205775] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.209522] PCI: Using configuration type 1 for base access [ 0.212118] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.221261] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.223055] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.226134] cryptd: max_cpu_qlen set to 1000 [ 0.227918] ACPI: Added _OSI(Module Device) [ 0.228000] ACPI: Added _OSI(Processor Device) [ 0.228000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.228023] ACPI: Added _OSI(Processor Aggregator Device) [ 0.234534] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.242764] ACPI: Interpreter enabled [ 0.244045] ACPI: PM: (supports S0 S3 S4 S5) [ 0.245007] ACPI: Using IOAPIC for interrupt routing [ 0.247069] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.251399] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.263215] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.265023] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.268011] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.272127] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.278526] acpiphp: Slot [2] registered [ 0.280121] acpiphp: Slot [5] registered [ 0.282092] acpiphp: Slot [6] registered [ 0.284104] acpiphp: Slot [7] registered [ 0.285073] acpiphp: Slot [8] registered [ 0.287103] acpiphp: Slot [9] registered [ 0.289067] acpiphp: Slot [10] registered [ 0.291059] acpiphp: Slot [3] registered [ 0.292067] acpiphp: Slot [4] registered [ 0.294092] acpiphp: Slot [11] registered [ 0.296113] acpiphp: Slot [12] registered [ 0.298068] acpiphp: Slot [13] registered [ 0.299064] acpiphp: Slot [14] registered [ 0.301060] acpiphp: Slot [15] registered [ 0.303177] acpiphp: Slot [16] registered [ 0.304071] acpiphp: Slot [17] registered [ 0.306066] acpiphp: Slot [18] registered [ 0.308123] acpiphp: Slot [19] registered [ 0.310061] acpiphp: Slot [20] registered [ 0.311064] acpiphp: Slot [21] registered [ 0.313083] acpiphp: Slot [22] registered [ 0.315176] acpiphp: Slot [23] registered [ 0.317221] acpiphp: Slot [24] registered [ 0.319142] acpiphp: Slot [25] registered [ 0.320056] acpiphp: Slot [26] registered [ 0.322073] acpiphp: Slot [27] registered [ 0.324061] acpiphp: Slot [28] registered [ 0.326078] acpiphp: Slot [29] registered [ 0.327063] acpiphp: Slot [30] registered [ 0.329066] acpiphp: Slot [31] registered [ 0.331065] PCI host bridge to bus 0000:00 [ 0.333012] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.335014] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.338018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.341014] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.344087] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.348025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.350149] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.353984] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.357075] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.368011] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.373047] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.378037] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.380040] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.383011] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.386339] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.389967] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.393095] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.396783] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.401010] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.410014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.413000] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.416047] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.422000] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.430014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.459013] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.476000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.488013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.500030] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.514016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.529158] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.539016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.550024] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.571019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.582446] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.589021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.595017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.612016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.626424] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.634017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.639011] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.656000] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.668568] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.681017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.691017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.711000] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.728157] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.731357] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.734615] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.737922] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.740214] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.746041] iommu: Default domain type: Passthrough [ 0.747485] SCSI subsystem initialized [ 0.749129] ACPI: bus type USB registered [ 0.750095] usbcore: registered new interface driver usbfs [ 0.752059] usbcore: registered new interface driver hub [ 0.754093] usbcore: registered new device driver usb [ 0.756158] pps_core: LinuxPPS API ver. 1 registered [ 0.757011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.759066] PTP clock support registered [ 0.761205] EDAC MC: Ver: 3.0.0 [ 0.762000] PCI: Using ACPI for IRQ routing [ 0.762979] NetLabel: Initializing [ 0.765009] NetLabel: domain hash size = 128 [ 0.766007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.769226] NetLabel: unlabeled traffic allowed by default [ 0.772318] vgaarb: loaded [ 0.773426] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.776012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.784223] clocksource: Switched to clocksource kvm-clock [ 0.929895] VFS: Disk quotas dquot_6.6.0 [ 0.931617] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.934061] *** VALIDATE ramfs *** [ 0.935187] *** VALIDATE hugetlbfs *** [ 0.937525] pnp: PnP ACPI init [ 0.940076] pnp: PnP ACPI: found 6 devices [ 0.955094] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.958341] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.961741] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.965121] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.967540] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.971072] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.978720] NET: Registered protocol family 2 [ 0.982884] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.994381] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.001395] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.009878] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.013867] TCP: Hash tables configured (established 65536 bind 65536) [ 1.017662] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.021219] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.026261] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.029870] NET: Registered protocol family 1 [ 1.032701] RPC: Registered named UNIX socket transport module. [ 1.035164] RPC: Registered udp transport module. [ 1.036968] RPC: Registered tcp transport module. [ 1.038369] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.040634] NET: Registered protocol family 44 [ 1.041771] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.044240] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.046658] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.049304] PCI: CLS 0 bytes, default 64 [ 1.051277] Unpacking initramfs... [ 2.647519] debug: unmapping init [mem 0xffff90a7fcc54000-0xffff90a7fffbffff] [ 2.653753] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.655819] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.659604] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.293819] Initialise system trusted keyrings [ 3.295399] Key type blacklist registered [ 3.297191] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.308830] zbud: loaded [ 3.311569] *** VALIDATE nfs *** [ 3.312716] *** VALIDATE nfs4 *** [ 3.314617] pstore: using deflate compression [ 3.319326] Platform Keyring initialized [ 3.438428] NET: Registered protocol family 38 [ 3.440578] Key type asymmetric registered [ 3.442319] Asymmetric key parser 'x509' registered [ 3.444216] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.447395] io scheduler mq-deadline registered [ 3.448949] io scheduler kyber registered [ 3.450304] io scheduler bfq registered [ 3.452577] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.455544] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.458408] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.461340] ACPI: Power Button [PWRF] [ 3.466707] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.473092] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.489306] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.498784] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.518829] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.554636] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.592956] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.598309] Non-volatile memory driver v1.3 [ 3.600264] Linux agpgart interface v0.103 [ 3.628639] virtio_blk virtio1: [vda] 145800 512-byte logical blocks (74.6 MB/71.2 MiB) [ 3.631065] vda: detected capacity change from 0 to 74649600 [ 3.647424] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.650597] vdb: detected capacity change from 0 to 1073741824 [ 3.665120] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.668181] vdc: detected capacity change from 0 to 2621440000 [ 3.682662] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.686101] vdd: detected capacity change from 0 to 2621440000 [ 3.701136] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.704268] vde: detected capacity change from 0 to 4294967296 [ 3.722734] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.726399] vdf: detected capacity change from 0 to 4294967296 [ 3.734514] libphy: Fixed MDIO Bus: probed [ 3.741639] usbcore: registered new interface driver usbserial_generic [ 3.744748] usbserial: USB Serial support registered for generic [ 3.747211] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.751193] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.752614] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.754628] mousedev: PS/2 mouse device common for all mice [ 3.757666] rtc_cmos 00:05: RTC can wake from S4 [ 3.759776] rtc_cmos 00:05: registered as rtc0 [ 3.761066] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.763578] intel_pstate: CPU model not supported [ 3.766176] hid: raw HID events driver (C) Jiri Kosina [ 3.767970] usbcore: registered new interface driver usbhid [ 3.770429] usbhid: USB HID core driver [ 3.772374] drop_monitor: Initializing network drop monitor service [ 3.773610] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.777747] Initializing XFRM netlink socket [ 3.778123] NET: Registered protocol family 10 [ 3.783094] Segment Routing with IPv6 [ 3.786172] NET: Registered protocol family 17 [ 3.787902] mpls_gso: MPLS GSO support [ 3.791205] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.798916] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.804267] RAS: Correctable Errors collector initialized. [ 3.806728] AVX version of gcm_enc/dec engaged. [ 3.808795] AES CTR mode by8 optimization enabled [ 3.921988] sched_clock: Marking stable (3921915699, 0)->(4900226801, -978311102) [ 3.925718] registered taskstats version 1 [ 3.927878] Loading compiled-in X.509 certificates [ 3.929975] zswap: loaded using pool lzo/zbud [ 3.984938] Key type big_key registered [ 4.007269] Key type encrypted registered [ 4.009129] ima: No TPM chip found, activating TPM-bypass! [ 4.011969] ima: Allocated hash algorithm: sha1 [ 4.013963] ima: No architecture policies found [ 4.016146] evm: Initialising EVM extended attributes: [ 4.018367] evm: security.selinux [ 4.019763] evm: security.ima [ 4.020982] evm: security.capability [ 4.022814] evm: HMAC attrs: 0x1 [ 4.025715] rtc_cmos 00:05: setting system clock to 2026-07-18 02:22:35 UTC (1784341355) [ 4.037216] debug: unmapping init [mem 0xffffffff85a03000-0xffffffff85bfffff] [ 4.041420] debug: unmapping init [mem 0xffffffff84782000-0xffffffff84a58fff] [ 4.051218] Write protecting the kernel read-only data: 28672k [ 4.056362] debug: unmapping init [mem 0xffffffff82e03000-0xffffffff82ffffff] [ 4.060739] debug: unmapping init [mem 0xffffffff83714000-0xffffffff837fffff] [ 4.110172] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.127871] systemd[1]: Detected virtualization kvm. [ 4.130175] systemd[1]: Detected architecture x86-64. [ 4.133321] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.164823] systemd[1]: No hostname configured. [ 4.167251] systemd[1]: Set hostname to . [ 4.170145] random: systemd: uninitialized urandom read (16 bytes read) [ 4.173493] systemd[1]: Initializing machine ID from random generator. [ 4.284628] random: ln: uninitialized urandom read (6 bytes read) [ 4.419315] random: systemd: uninitialized urandom read (16 bytes read) [ 4.422179] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.429930] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.436110] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Control Socket. Starting Setup Virtual Console... [ OK ] Reached target Swap. Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.523975] device-mapper: uevent: version 1.0.3 [ 5.526429] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 6.853120] virtio_net virtio0 ens2: renamed from eth0 [ 6.938726] random: fast init done [ 6.958227] scsi host0: ata_piix [ 7.029287] scsi host1: ata_piix [ 7.034880] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.041790] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 12.575943] random: crng init done [ 12.577646] random: 7 urandom warning(s) missed due to ratelimiting [ 12.742397] dracut-initqueue[586]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 14.411889] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Swap. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.900933] printk: systemd: 26 output lines suppressed due to ratelimiting [ 16.297726] SELinux: Disabled at runtime. [ 16.379492] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 16.390764] systemd[1]: Detected virtualization kvm. [ 16.392866] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 17.462956] systemd[1]: initrd-switch-root.service: Succeeded. [ 17.467157] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 17.480487] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 17.484320] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 17.487767] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 17.496312] systemd[1]: Starting Journal Service... Starting Journal Service... [ 17.503126] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... [ 17.562632] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 18.461114] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 19.175753] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 19.258798] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 19.474116] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 19.506527] EDAC sbridge: Ver: 1.1.2 [ 22.053263] Key type dns_resolver registered [ 22.437646] NFS: Registering the id_resolver key type [ 22.439876] Key type id_resolver registered [ 22.441410] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg349-server login: [ 49.686419] libcfs: loading out-of-tree module taints kernel. [ 49.705806] Key type ._llcrypt registered [ 49.707476] Key type .llcrypt registered [ 49.743784] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 56.589084] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 57.184427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 57.190850] alg: No test for adler32 (adler32-zlib) [ 58.137537] Lustre: Lustre: Build Version: 2.17.54_173_gd3f3fcc [ 58.398848] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 59.999138] Key type lgssc registered [ 60.500093] Lustre: Echo OBD driver; http://www.lustre.org/ [ 66.547183] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 79.919946] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 84.181649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 84.189833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 85.275634] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 85.288349] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 85.332479] Lustre: lustre-MDT0000: new disk, initializing [ 85.358786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 85.367855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 86.676752] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 91.869846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 91.901013] Lustre: 6482:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 91.911380] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 91.914353] Lustre: Skipped 1 previous similar message [ 91.951840] Lustre: lustre-MDT0001: new disk, initializing [ 91.969220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 91.976468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 91.980812] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 93.252504] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 95.491907] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 98.470226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 98.555764] Lustre: lustre-OST0000: new disk, initializing [ 98.557646] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 98.560467] Lustre: 8418:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 98.576172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 99.754713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 99.759180] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 99.775219] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 100.379936] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 105.847280] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 105.899931] Lustre: lustre-OST0001: new disk, initializing [ 105.901751] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 105.903957] Lustre: 9495:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 105.924055] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 106.431351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 106.435176] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 106.456417] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 107.942529] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 113.726064] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 120.309905] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 122.275210] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing check_logdir /tmp/testlogs/ [ 123.724395] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing yml_node [ 125.166547] Lustre: DEBUG MARKER: Client: 2.17.54.173 [ 126.059532] Lustre: DEBUG MARKER: MDS: 2.17.54.173 [ 126.888500] Lustre: DEBUG MARKER: OSS: 2.17.54.173 [ 127.398538] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Jul 17 22:24:38 EDT 2026 [ 132.560490] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 133.019877] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 147.423976] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 147.424375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 147.429107] Lustre: Skipped 3 previous similar messages [ 147.431023] Lustre: Skipped 2 previous similar messages [ 150.868813] Lustre: server umount lustre-MDT0000 complete [ 152.038584] LustreError: 6476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784341503 with bad export cookie 12283321335219232620 [ 152.039942] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 152.042505] LustreError: 6476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 152.543611] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 152.543929] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 152.549129] Lustre: Skipped 1 previous similar message [ 157.663559] LustreError: 10070:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 157.663606] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 157.668659] LustreError: 10070:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 157.674655] Lustre: Skipped 2 previous similar messages [ 161.759773] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 161.759774] LustreError: 6491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 161.767835] LustreError: 6491:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 166.367112] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 166.431675] Lustre: server umount lustre-MDT0001 complete [ 173.690719] Lustre: server umount lustre-OST0000 complete [ 180.941181] Lustre: server umount lustre-OST0001 complete [ 182.910977] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 185.688903] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 196.641896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 196.716136] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 196.728251] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 196.764608] Lustre: lustre-MDT0000: new disk, initializing [ 196.783688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 196.790478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 197.990915] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 202.021089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 202.059041] Lustre: 15035:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 202.158646] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 203.341487] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 206.790559] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 208.347792] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 210.292116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 210.382243] Lustre: lustre-OST0000: new disk, initializing [ 210.383425] Lustre: Skipped 1 previous similar message [ 210.384785] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 210.386594] Lustre: Skipped 2 previous similar messages [ 210.388303] Lustre: 17001:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 210.414429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 210.416396] Lustre: Skipped 1 previous similar message [ 212.109715] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 212.278196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 212.280462] Lustre: Skipped 1 previous similar message [ 212.282326] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 212.316395] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 215.489579] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 216.855652] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 216.926688] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 218.133226] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 218.197917] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 222.688802] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 222.690532] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 222.691765] LustreError: Skipped 1 previous similar message [ 222.697853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 222.700649] Lustre: Skipped 2 previous similar messages [ 224.840869] Lustre: server umount lustre-OST0000 complete [ 227.809470] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 227.809756] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 227.816200] Lustre: Skipped 2 previous similar messages [ 232.092317] Lustre: server umount lustre-MDT0000 complete [ 232.928925] LustreError: 15047:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 232.934439] LustreError: 15047:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 233.198930] LustreError: 15030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784341584 with bad export cookie 12283321335219234139 [ 233.202099] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 233.202396] LustreError: 15030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 233.305665] Lustre: server umount lustre-MDT0001 complete [ 234.341871] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 22:26:25 (1784341585) [ 236.420372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 236.576850] LustreError: 19262:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 236.598672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 237.833404] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 240.577042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 241.947410] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 244.278379] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 246.066481] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 248.247173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 250.235167] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 252.510371] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 266.720760] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 266.721179] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 266.725203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 266.728695] Lustre: Skipped 1 previous similar message [ 266.730993] Lustre: Skipped 2 previous similar messages [ 267.532173] Lustre: server umount lustre-OST0000 complete [ 270.878027] Lustre: server umount lustre-MDT0000 complete [ 271.327760] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 271.331405] LustreError: Skipped 1 previous similar message [ 271.334199] LustreError: 19262:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 272.163649] LustreError: 19244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784341623 with bad export cookie 12283321335219235021 [ 272.168825] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 272.170351] LustreError: 19244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 272.284388] Lustre: server umount lustre-MDT0001 complete [ 275.872707] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 276.838831] Key type lgssc unregistered [ 276.972401] LNet: 22813:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 276.975799] LNetError: 22813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 276.984338] LNet: Removed LNI 192.168.203.149@tcp [ 277.319124] Key type .llcrypt unregistered [ 277.320220] Key type ._llcrypt unregistered [ 285.695672] Key type ._llcrypt registered [ 285.696779] Key type .llcrypt registered [ 285.731683] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 286.002841] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 286.009113] alg: No test for adler32 (adler32-zlib) [ 286.860941] Lustre: Lustre: Build Version: 2.17.54_173_gd3f3fcc [ 286.951437] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 288.527164] Key type lgssc registered [ 288.897396] Lustre: Echo OBD driver; http://www.lustre.org/ [ 291.449128] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 291.455242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 292.584628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 293.885337] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 296.644212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 296.771343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 298.095464] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 300.283717] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 301.932343] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 304.087426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 304.203831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 306.077529] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 308.340943] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 317.541081] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 323.041079] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 323.043042] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 323.043894] LustreError: Skipped 1 previous similar message [ 323.049761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 323.053651] Lustre: Skipped 1 previous similar message [ 328.160654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 328.163575] Lustre: Skipped 1 previous similar message [ 333.282244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 333.284991] Lustre: Skipped 1 previous similar message [ 337.375108] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 337.425592] Lustre: server umount lustre-OST0000 complete [ 343.007853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 343.011232] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 343.016164] Lustre: Skipped 1 previous similar message [ 343.018662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 344.789812] Lustre: server umount lustre-MDT0000 complete [ 345.952310] LustreError: 24181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784341697 with bad export cookie 11998241139008932052 [ 345.955473] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 345.958305] LustreError: 24181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 346.074503] Lustre: server umount lustre-MDT0001 complete [ 349.785810] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 350.864051] Key type lgssc unregistered [ 350.996388] LNet: 28195:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 351.000225] LNetError: 28195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 351.007341] LNet: Removed LNI 192.168.203.149@tcp [ 351.349649] Key type .llcrypt unregistered [ 351.351100] Key type ._llcrypt unregistered [ 360.261670] Key type ._llcrypt registered [ 360.262938] Key type .llcrypt registered [ 360.303716] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 360.724065] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 360.731698] alg: No test for adler32 (adler32-zlib) [ 361.584081] Lustre: Lustre: Build Version: 2.17.54_173_gd3f3fcc [ 361.671885] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 363.255121] Key type lgssc registered [ 363.599894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 365.960358] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 365.965587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 367.088273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 368.283726] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 370.866183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 370.983807] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 372.239190] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 374.440601] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 376.205188] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 380.424315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 380.527620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 382.386867] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 384.798510] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 391.781551] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 397.282930] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 397.285347] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 397.287085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 397.291598] Lustre: Skipped 1 previous similar message [ 398.795596] Lustre: server umount lustre-OST0000 complete [ 401.888203] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 401.890686] LustreError: Skipped 1 previous similar message [ 401.892556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 401.897672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 401.899847] Lustre: Skipped 1 previous similar message [ 402.136517] Lustre: server umount lustre-MDT0000 complete [ 402.400500] LustreError: 30322:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 402.406153] LustreError: 30322:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 403.302167] LustreError: 29566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784341754 with bad export cookie 12493653923190463720 [ 403.307096] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 403.412720] Lustre: server umount lustre-MDT0001 complete [ 406.999258] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 408.057592] Key type lgssc unregistered [ 408.180330] LNet: 33479:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 408.183949] LNetError: 33479:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 408.192463] LNet: Removed LNI 192.168.203.149@tcp [ 408.481839] Key type .llcrypt unregistered [ 408.483288] Key type ._llcrypt unregistered [ 417.222333] Key type ._llcrypt registered [ 417.223812] Key type .llcrypt registered [ 417.261669] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 417.579887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 417.614248] alg: No test for adler32 (adler32-zlib) [ 418.484981] Lustre: Lustre: Build Version: 2.17.54_173_gd3f3fcc [ 418.571058] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 420.151143] Key type lgssc registered [ 420.520569] Lustre: Echo OBD driver; http://www.lustre.org/ [ 423.056583] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 423.062902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 424.195558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 425.532216] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 428.273451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 428.397165] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 429.804321] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 432.239633] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 434.000978] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 436.138889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 436.248855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 438.109939] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 440.489745] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 449.578784] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 454.624951] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 454.624954] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 454.626216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 454.637213] Lustre: Skipped 1 previous similar message [ 456.078662] Lustre: server umount lustre-OST0000 complete [ 459.231842] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 459.234583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 459.238311] Lustre: Skipped 1 previous similar message [ 459.239980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 459.542180] Lustre: server umount lustre-MDT0000 complete [ 459.745209] LustreError: 34864:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 459.751629] LustreError: 34864:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 460.785085] LustreError: 34848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784341812 with bad export cookie 2273940087513128240 [ 460.787124] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 460.789138] LustreError: 34848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 460.898260] Lustre: server umount lustre-MDT0001 complete [ 464.457743] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 465.455782] Key type lgssc unregistered [ 465.580298] LNet: 38420:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 465.583885] LNetError: 38420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 465.592298] LNet: Removed LNI 192.168.203.149@tcp [ 465.902743] Key type .llcrypt unregistered [ 465.903832] Key type ._llcrypt unregistered [ 474.104890] Key type ._llcrypt registered [ 474.105934] Key type .llcrypt registered [ 474.141286] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 474.518325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 474.548739] alg: No test for adler32 (adler32-zlib) [ 475.388920] Lustre: Lustre: Build Version: 2.17.54_173_gd3f3fcc [ 475.461800] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 477.039154] Key type lgssc registered [ 477.381397] Lustre: Echo OBD driver; http://www.lustre.org/ [ 479.870452] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 479.877285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 481.023122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 482.218988] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 484.916281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 485.035570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 486.321973] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 488.561874] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 490.319561] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 492.519440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 492.632588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 494.570926] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 496.922196] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 505.954731] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 510.431785] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 510.434838] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 510.440791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 511.457257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 511.458739] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 511.464271] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 512.701852] Lustre: lustre-OST0000: Not available for connect from 192.168.203.49@tcp (stopping) [ 512.704015] Lustre: Skipped 1 previous similar message [ 516.576197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 516.578971] Lustre: Skipped 1 previous similar message [ 521.696310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 521.698349] Lustre: Skipped 2 previous similar messages [ 524.255119] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 524.307096] Lustre: server umount lustre-OST0000 complete [ 541.663801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 541.666382] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 541.670874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 541.672499] Lustre: Skipped 1 previous similar message [ 543.377204] Lustre: server umount lustre-MDT0000 complete [ 544.640315] LustreError: 39791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784341896 with bad export cookie 8754765920062395319 [ 544.643222] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 544.645544] LustreError: 39791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 550.826514] Lustre: server umount lustre-MDT0001 complete [ 554.150677] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 555.059940] Key type lgssc unregistered [ 555.175320] LNet: 43534:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 555.177572] LNetError: 43534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 555.185233] LNet: Removed LNI 192.168.203.149@tcp [ 555.458955] Key type .llcrypt unregistered [ 555.460198] Key type ._llcrypt unregistered [ 563.703400] Key type ._llcrypt registered [ 563.704745] Key type .llcrypt registered [ 563.737379] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 564.051532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 564.095768] alg: No test for adler32 (adler32-zlib) [ 564.946462] Lustre: Lustre: Build Version: 2.17.54_173_gd3f3fcc [ 565.024992] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 566.599098] Key type lgssc registered [ 566.971652] Lustre: Echo OBD driver; http://www.lustre.org/ [ 569.495655] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 569.503710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 570.641040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 571.880233] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 574.564786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 574.688932] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 576.011505] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 578.232679] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 579.866033] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 581.883364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 581.981129] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 583.706609] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 585.996177] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 596.328947] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 600.544213] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 600.546977] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 600.551339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 601.568276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 601.569641] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 601.574106] Lustre: Skipped 1 previous similar message [ 605.009623] Lustre: server umount lustre-MDT0000 complete [ 606.212037] LustreError: 44948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784341957 with bad export cookie 14220932615181683880 [ 606.215741] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 606.217080] LustreError: 44948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 606.688029] LustreError: 45651:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 606.688252] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 606.693910] LustreError: 45651:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 606.698868] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 606.703592] Lustre: Skipped 3 previous similar messages [ 611.807963] LustreError: 45651:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 611.808278] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 611.814626] LustreError: 45651:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 611.816779] Lustre: Skipped 1 previous similar message [ 612.391462] Lustre: server umount lustre-MDT0001 complete [ 630.164470] Lustre: server umount lustre-OST0000 complete [ 635.421766] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 636.446372] Key type lgssc unregistered [ 636.575326] LNet: 48801:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 636.578541] LNetError: 48801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 636.587413] LNet: Removed LNI 192.168.203.149@tcp [ 636.890960] Key type .llcrypt unregistered [ 636.892840] Key type ._llcrypt unregistered [ 646.562099] Key type ._llcrypt registered [ 646.564044] Key type .llcrypt registered [ 646.602704] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 647.035799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 647.043861] alg: No test for adler32 (adler32-zlib) [ 647.896950] Lustre: Lustre: Build Version: 2.17.54_173_gd3f3fcc [ 647.986670] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 649.567145] Key type lgssc registered [ 649.944431] Lustre: Echo OBD driver; http://www.lustre.org/ [ 652.564633] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 652.573390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 667.999289] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 673.119252] LustreError: 50165:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.149@tcp: failed processing log, type 4: rc = -110 [ 699.743185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 701.412371] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 703.488260] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 867.292984] Lustre: server umount lustre-OST0000 complete [ 872.281792] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 873.219427] Key type lgssc unregistered [ 873.333329] LNet: 51838:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 873.336234] LNetError: 51838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 873.345267] LNet: Removed LNI 192.168.203.149@tcp [ 873.623348] Key type .llcrypt unregistered [ 873.624591] Key type ._llcrypt unregistered [ 881.769582] Key type ._llcrypt registered [ 881.770804] Key type .llcrypt registered [ 881.803570] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 882.102880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 882.143080] alg: No test for adler32 (adler32-zlib) [ 882.986716] Lustre: Lustre: Build Version: 2.17.54_173_gd3f3fcc [ 883.068390] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 884.647114] Key type lgssc registered [ 884.967026] Lustre: Echo OBD driver; http://www.lustre.org/ [ 887.268144] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 887.273191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 888.397380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 889.603713] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 892.225406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 892.335246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 893.557930] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 895.836590] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 897.497050] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 899.524284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 899.624936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 901.406647] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 903.513329] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 906.720624] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 906.725026] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 906.731994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 910.278645] Lustre: server umount lustre-OST0000 complete [ 911.840150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 911.840692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 911.843429] Lustre: Skipped 1 previous similar message [ 916.960170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 916.962206] Lustre: Skipped 2 previous similar messages [ 917.519614] Lustre: server umount lustre-MDT0000 complete [ 918.639955] LustreError: 53962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784342270 with bad export cookie 5973549283674285640 [ 918.642780] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 918.644538] LustreError: 53962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 918.743775] Lustre: server umount lustre-MDT0001 complete [ 922.090136] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 923.019634] Key type lgssc unregistered [ 923.140313] LNet: 56753:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 923.143474] LNetError: 56753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 923.151259] LNet: Removed LNI 192.168.203.149@tcp [ 923.450757] Key type .llcrypt unregistered [ 923.452083] Key type ._llcrypt unregistered [ 931.306248] Key type ._llcrypt registered [ 931.307444] Key type .llcrypt registered [ 931.340713] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 931.723286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 931.731990] alg: No test for adler32 (adler32-zlib) [ 932.568756] Lustre: Lustre: Build Version: 2.17.54_173_gd3f3fcc [ 932.648775] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 934.223138] Key type lgssc registered [ 934.545823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 936.854283] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 936.860079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 952.287411] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 957.407305] LustreError: 58116:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.149@tcp: failed processing log, type 4: rc = -110 [ 984.031177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 985.744606] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 987.816926] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 989.876203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 990.025379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 990.044197] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 991.319783] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 994.122293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 994.251988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 995.537929] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 997.879042] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 999.713780] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1000.415726] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1000.418617] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1000.424144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1004.516979] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1004.518552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1004.521344] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1006.538264] Lustre: server umount lustre-OST0000 complete [ 1020.505153] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1020.509803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1020.511947] Lustre: Skipped 2 previous similar messages [ 1024.608402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1024.610422] Lustre: Skipped 1 previous similar message [ 1026.708322] Lustre: server umount lustre-MDT0000 complete [ 1027.943138] LustreError: 59181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784342379 with bad export cookie 560078592883833005 [ 1027.946825] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1027.947906] LustreError: 59181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1034.093422] Lustre: server umount lustre-MDT0001 complete [ 1037.747046] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 1038.821257] Key type lgssc unregistered [ 1038.942395] LNet: 61859:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1038.944899] LNetError: 61859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1038.957351] LNet: Removed LNI 192.168.203.149@tcp [ 1039.277522] Key type .llcrypt unregistered [ 1039.278776] Key type ._llcrypt unregistered [ 1047.849205] Key type ._llcrypt registered [ 1047.850683] Key type .llcrypt registered [ 1047.889018] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 1048.284695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1048.293892] alg: No test for adler32 (adler32-zlib) [ 1049.143026] Lustre: Lustre: Build Version: 2.17.54_173_gd3f3fcc [ 1049.226143] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 1050.807130] Key type lgssc registered [ 1051.165238] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1053.731440] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1053.741664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1054.896110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1056.408527] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1059.114136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1059.231520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1060.537563] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1062.878863] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1064.701362] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1067.013199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1067.129347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1068.991697] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1071.318224] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1080.422951] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1084.896447] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1084.901178] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1084.906894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1085.921220] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1085.922582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1085.923181] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1085.929062] Lustre: Skipped 1 previous similar message [ 1089.868617] Lustre: server umount lustre-OST0000 complete [ 1095.136079] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1095.138319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1095.142174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1101.280299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1101.282283] Lustre: Skipped 3 previous similar messages [ 1105.375156] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1105.438423] Lustre: server umount lustre-MDT0000 complete [ 1106.399967] LustreError: 63248:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1106.404185] LustreError: 63248:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1106.641137] LustreError: 63228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784342458 with bad export cookie 7196062011568384299 [ 1106.644437] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1106.647355] LustreError: 63228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1106.766247] Lustre: server umount lustre-MDT0001 complete [ 1110.484694] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 1111.479913] Key type lgssc unregistered [ 1111.624478] LNet: 66799:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1111.627321] LNetError: 66799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1111.635274] LNet: Removed LNI 192.168.203.149@tcp [ 1111.931182] Key type .llcrypt unregistered [ 1111.932324] Key type ._llcrypt unregistered [ 1121.320158] Key type ._llcrypt registered [ 1121.322010] Key type .llcrypt registered [ 1121.361924] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 1121.697491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1121.747370] alg: No test for adler32 (adler32-zlib) [ 1122.599011] Lustre: Lustre: Build Version: 2.17.54_173_gd3f3fcc [ 1122.689103] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 1124.271173] Key type lgssc registered [ 1124.644924] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1127.587446] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 22:41:18 (1784342478) [ 1129.738165] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1129.743759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1130.889163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1132.189132] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1134.976141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1135.113987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1136.482699] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1138.929854] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1140.660648] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1142.814038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1142.925612] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1144.788376] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1147.110696] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1156.262690] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1159.966326] Lustre: Failing over lustre-MDT0000 [ 1160.143631] Lustre: server umount lustre-MDT0000 complete [ 1160.672278] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1160.675802] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1160.682571] LustreError: 69908:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1160.687125] LustreError: 69908:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1161.696686] LustreError: 69215:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1161.697738] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1162.944772] LustreError: 68473:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1162.951524] LustreError: 68473:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1163.225759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1163.264847] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1163.318876] Lustre: *** cfs_fail_loc=135, val=0*** [ 1163.360674] LustreError: 71586:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 1163.363660] LustreError: 71586:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.149@tcp: cfg command failed: rc = -2 [ 1163.367234] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1163.367234] [ 1163.371688] LustreError: MGC192.168.203.149@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1163.375958] LustreError: 71574:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1163.378769] LustreError: 71574:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 1163.381451] LustreError: 71574:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 1163.383310] LustreError: 71574:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1163.394042] Lustre: server umount lustre-MDT0000 complete [ 1163.395899] LustreError: 71574:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1165.494080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1165.532845] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1165.622807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1165.644072] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1166.863628] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1168.055172] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1170.918271] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1170.930797] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 1170.946600] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1176.038415] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1176.040121] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1176.040588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1176.040612] LustreError: Skipped 1 previous similar message [ 1176.043934] Lustre: Skipped 2 previous similar messages [ 1176.049932] Lustre: Skipped 1 previous similar message [ 1177.997445] Lustre: server umount lustre-OST0000 complete [ 1181.153045] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1181.153962] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1181.154792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1181.161522] Lustre: Skipped 1 previous similar message [ 1185.301289] Lustre: server umount lustre-MDT0000 complete [ 1186.271935] LustreError: 69215:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1186.277704] LustreError: 69215:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1186.540207] LustreError: 68459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784342538 with bad export cookie 3526535890471466281 [ 1186.544180] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1186.544955] LustreError: 68459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1186.651526] Lustre: server umount lustre-MDT0001 complete [ 1190.254718] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 1191.252524] Key type lgssc unregistered [ 1191.381505] LNet: 73357:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1191.385800] LNetError: 73357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1191.401321] LNet: Removed LNI 192.168.203.149@tcp [ 1191.697895] Key type .llcrypt unregistered [ 1191.699730] Key type ._llcrypt unregistered [ 1200.316375] Key type ._llcrypt registered [ 1200.317532] Key type .llcrypt registered [ 1200.352018] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 1200.708097] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1200.737458] alg: No test for adler32 (adler32-zlib) [ 1201.592305] Lustre: Lustre: Build Version: 2.17.54_173_gd3f3fcc [ 1201.676897] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 1203.255156] Key type lgssc registered [ 1203.620915] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1206.188174] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1206.193983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1207.342742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1208.765849] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1211.747057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1211.879499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1213.209486] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1215.546472] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1217.374726] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1219.591345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1219.708988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1221.613520] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1224.043949] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1232.037829] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1236.769471] Lustre: Failing over lustre-MDT0000 [ 1236.992611] Lustre: server umount lustre-MDT0000 complete [ 1237.473920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1237.474214] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1237.478741] Lustre: Skipped 1 previous similar message [ 1237.479062] LustreError: 74742:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1237.479068] LustreError: 74742:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1239.742375] LustreError: 74773:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1239.748208] LustreError: 74773:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1240.039028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1240.079622] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1240.137965] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1240.194413] LustreError: 77855:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 1240.197296] LustreError: 77855:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.149@tcp: cfg command failed: rc = -9 [ 1240.199864] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1240.199864] [ 1240.203889] LustreError: MGC192.168.203.149@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1240.211088] LustreError: 77843:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1240.215274] LustreError: 77843:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 1240.220028] LustreError: 77843:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 1240.223566] LustreError: 77843:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1240.234653] Lustre: server umount lustre-MDT0000 complete [ 1240.236060] LustreError: 77843:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1242.422365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1242.467486] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1242.557526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1242.586518] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1243.843243] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1244.854505] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1247.720120] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1247.728986] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1247.744246] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1252.833640] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1252.834377] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1252.836511] LustreError: Skipped 1 previous similar message [ 1252.842100] Lustre: Skipped 1 previous similar message [ 1252.844588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1252.847425] Lustre: Skipped 1 previous similar message [ 1254.795750] Lustre: server umount lustre-OST0000 complete [ 1257.952884] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1257.953077] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1257.953505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1257.961882] Lustre: Skipped 1 previous similar message [ 1262.165131] Lustre: server umount lustre-MDT0000 complete [ 1263.071962] LustreError: 74747:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1263.076478] LustreError: 74747:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1263.384366] LustreError: 74774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784342614 with bad export cookie 12591810864355560078 [ 1263.387311] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1263.389081] LustreError: 74774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1263.505666] Lustre: server umount lustre-MDT0001 complete [ 1267.013638] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 1267.991389] Key type lgssc unregistered [ 1268.108385] LNet: 79631:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1268.111122] LNetError: 79631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1268.121297] LNet: Removed LNI 192.168.203.149@tcp [ 1268.423059] Key type .llcrypt unregistered [ 1268.424185] Key type ._llcrypt unregistered [ 1272.417975] Key type ._llcrypt registered [ 1272.419046] Key type .llcrypt registered [ 1272.452976] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 1272.547293] Key type .llcrypt unregistered [ 1272.548620] Key type ._llcrypt unregistered [ 1281.391824] Key type ._llcrypt registered [ 1281.392969] Key type .llcrypt registered [ 1281.432115] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 1281.846686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1281.873534] alg: No test for adler32 (adler32-zlib) [ 1282.750579] Lustre: Lustre: Build Version: 2.17.54_173_gd3f3fcc [ 1282.848463] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 1284.431200] Key type lgssc registered [ 1284.847271] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1287.416869] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1287.422381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1288.557665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1289.893445] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1292.572612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1292.686781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1294.005302] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1296.582274] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1298.555298] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1300.995173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1301.105589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1303.120152] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1305.792853] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1314.409966] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1319.906467] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1319.907477] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1319.907934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1319.911967] Lustre: Skipped 1 previous similar message [ 1325.025344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1325.028345] Lustre: Skipped 1 previous similar message [ 1325.325422] Lustre: server umount lustre-OST0000 complete [ 1328.607970] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1328.612462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1328.616677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1330.145403] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1332.695414] Lustre: server umount lustre-MDT0000 complete [ 1333.898360] LustreError: 81489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784342685 with bad export cookie 9794838106183548598 [ 1333.900789] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1333.903326] LustreError: 81489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1334.014376] Lustre: server umount lustre-MDT0001 complete [ 1337.430127] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 1338.388551] Key type lgssc unregistered [ 1338.512428] LNet: 85058:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1338.516388] LNetError: 85058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1338.527297] LNet: Removed LNI 192.168.203.149@tcp [ 1338.824514] Key type .llcrypt unregistered [ 1338.826392] Key type ._llcrypt unregistered [ 1347.214633] Key type ._llcrypt registered [ 1347.215733] Key type .llcrypt registered [ 1347.253345] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 1347.553411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1347.562388] alg: No test for adler32 (adler32-zlib) [ 1348.401596] Lustre: Lustre: Build Version: 2.17.54_173_gd3f3fcc [ 1348.483713] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 1350.055130] Key type lgssc registered [ 1350.408447] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1352.880200] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1352.886857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1354.041116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1355.372339] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1358.183450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1358.315175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1359.704594] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1362.126573] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1363.994256] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1366.194480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1366.310241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1368.188658] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1370.514139] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1378.597601] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 1383.905458] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1383.908211] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1383.909094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1383.912584] Lustre: Skipped 1 previous similar message [ 1385.291962] Lustre: server umount lustre-OST0000 complete [ 1388.778773] Lustre: server umount lustre-MDT0000 complete [ 1389.024497] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1389.024987] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1389.026840] LustreError: Skipped 1 previous similar message [ 1389.027273] LustreError: 86447:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1389.033427] Lustre: Skipped 1 previous similar message [ 1389.044477] LustreError: 86447:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1389.967229] LustreError: 86427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784342741 with bad export cookie 12014081063582841481 [ 1389.970602] LustreError: 86427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1389.971031] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1390.075912] Lustre: server umount lustre-MDT0001 complete [ 1393.751658] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 1394.741844] Key type lgssc unregistered [ 1394.880407] LNet: 89994:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1394.882818] LNetError: 89994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1394.891333] LNet: Removed LNI 192.168.203.149@tcp [ 1395.203750] Key type .llcrypt unregistered [ 1395.205053] Key type ._llcrypt unregistered [ 1403.901170] Key type ._llcrypt registered [ 1403.902274] Key type .llcrypt registered [ 1403.940424] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 1404.398924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1404.408965] alg: No test for adler32 (adler32-zlib) [ 1405.255833] Lustre: Lustre: Build Version: 2.17.54_173_gd3f3fcc [ 1405.340496] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 1406.919108] Key type lgssc registered [ 1407.257531] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1409.771238] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1409.779379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1410.927906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1412.260979] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1415.020052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1415.146126] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1416.590712] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1419.117483] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1421.024101] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1423.234219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1423.346054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1425.228946] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1427.624835] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1435.623936] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 1445.857961] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1445.859633] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1445.861253] LustreError: Skipped 1 previous similar message [ 1445.866544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1445.869926] Lustre: Skipped 1 previous similar message [ 1447.503689] Lustre: server umount lustre-OST0000 complete [ 1450.977535] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1450.977608] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1450.978184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1450.981387] Lustre: Skipped 2 previous similar messages [ 1454.812408] Lustre: server umount lustre-MDT0000 complete [ 1456.015133] LustreError: 91410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784342807 with bad export cookie 1496991909702890659 [ 1456.019206] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1456.022492] LustreError: 91410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1456.137958] Lustre: server umount lustre-MDT0001 complete [ 1459.640520] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 1460.640339] Key type lgssc unregistered [ 1460.761334] LNet: 94933:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1460.763838] LNetError: 94933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1460.772330] LNet: Removed LNI 192.168.203.149@tcp [ 1461.056157] Key type .llcrypt unregistered [ 1461.057298] Key type ._llcrypt unregistered [ 1469.612698] Key type ._llcrypt registered [ 1469.614042] Key type .llcrypt registered [ 1469.648611] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 1469.957293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1469.990653] alg: No test for adler32 (adler32-zlib) [ 1470.856299] Lustre: Lustre: Build Version: 2.17.54_173_gd3f3fcc [ 1470.966646] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 1472.559193] Key type lgssc registered [ 1472.966545] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1475.696920] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1475.704296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1491.167472] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1496.287408] LustreError: 96296:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.149@tcp: failed processing log, type 4: rc = -110 [ 1522.911200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1524.819781] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1527.219679] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1529.119511] Lustre: server umount lustre-OST0000 complete [ 1531.959780] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 22:48:02 (1784342882) [ 1534.286339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1534.441295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1535.843895] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1538.749490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1538.866593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1540.214987] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1542.911972] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1544.981606] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1547.134163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1547.255107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1549.084570] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1551.455192] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1559.528460] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 1569.761537] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1569.761632] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1569.765371] LustreError: Skipped 1 previous similar message [ 1569.770398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1569.774343] Lustre: Skipped 1 previous similar message [ 1571.154894] Lustre: server umount lustre-OST0000 complete [ 1574.880868] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1574.881245] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1574.881986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1574.890598] Lustre: Skipped 2 previous similar messages [ 1578.587965] Lustre: server umount lustre-MDT0000 complete [ 1579.831511] LustreError: 100601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784342931 with bad export cookie 11072023231378530000 [ 1579.834929] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1579.837014] LustreError: 100601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1579.954377] Lustre: server umount lustre-MDT0001 complete [ 1583.801134] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 1584.956622] Key type lgssc unregistered [ 1585.093446] LNet: 101640:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1585.095724] LNetError: 101640:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1585.102436] LNet: Removed LNI 192.168.203.149@tcp [ 1585.458936] Key type .llcrypt unregistered [ 1585.460761] Key type ._llcrypt unregistered [ 1600.103100] Key type ._llcrypt registered [ 1600.104370] Key type .llcrypt registered [ 1600.141283] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 1600.556477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1600.593890] alg: No test for adler32 (adler32-zlib) [ 1601.435460] Lustre: Lustre: Build Version: 2.17.54_173_gd3f3fcc [ 1601.508724] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 1603.079160] Key type lgssc registered [ 1603.403021] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1605.970184] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1605.975302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1607.102931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1608.400231] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1611.120349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1611.252609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1612.615338] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1615.037712] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1616.854783] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1619.105552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1619.236447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1621.049259] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1623.453486] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1632.550639] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 1636.832323] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1636.834756] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1636.838607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1637.858666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1637.859090] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1637.859138] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1638.543959] Lustre: server umount lustre-OST0000 complete [ 1641.831010] Lustre: server umount lustre-MDT0000 complete [ 1641.951711] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1641.953861] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1641.957520] LustreError: 103817:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1641.961616] LustreError: 103817:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1642.975961] LustreError: 103798:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1642.981274] LustreError: 103798:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1643.035299] LustreError: 103779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784342994 with bad export cookie 4306519474546033355 [ 1643.038187] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1643.038972] LustreError: 103779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1643.151746] Lustre: server umount lustre-MDT0001 complete [ 1646.641358] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 1647.561406] Key type lgssc unregistered [ 1647.685391] LNet: 107348:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1647.687716] LNetError: 107348:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1647.695315] LNet: Removed LNI 192.168.203.149@tcp [ 1648.009621] Key type .llcrypt unregistered [ 1648.011228] Key type ._llcrypt unregistered [ 1655.202443] Key type ._llcrypt registered [ 1655.203533] Key type .llcrypt registered [ 1655.238893] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 1655.621863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1655.641625] alg: No test for adler32 (adler32-zlib) [ 1656.491109] Lustre: Lustre: Build Version: 2.17.54_173_gd3f3fcc [ 1656.579722] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 1658.159291] Key type lgssc registered [ 1658.517786] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1661.236541] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1661.249789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1676.703300] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1681.823236] LustreError: 108566:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.149@tcp: failed processing log, type 4: rc = -110 [ 1708.447156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1710.455310] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1712.851730] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1715.005074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1715.101672] LustreError: MGC192.168.203.149@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1715.105856] LustreError: 109590:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1715.108282] LustreError: 109590:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 1715.111485] LustreError: 109590:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 1715.138301] Lustre: server umount lustre-MDT0000 complete [ 1715.139664] LustreError: 109590:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1728.000472] Lustre: server umount lustre-OST0000 complete [ 1730.988884] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 1734.100668] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 1745.388024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1745.485354] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1745.496367] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1745.535949] Lustre: lustre-MDT0000: new disk, initializing [ 1745.558333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1745.564163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1746.938584] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1751.048727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1751.089667] Lustre: 112375:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1751.114239] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1751.116248] Lustre: Skipped 1 previous similar message [ 1751.151121] Lustre: lustre-MDT0001: new disk, initializing [ 1751.182545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1751.193588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1751.197714] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1752.504303] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1756.281217] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1758.112646] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1760.315445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1760.433836] Lustre: lustre-OST0000: new disk, initializing [ 1760.435493] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1760.437926] Lustre: 114343:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1760.468721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1762.391962] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1763.449236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1763.453604] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1763.485787] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1765.934539] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1767.213646] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1767.281989] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1768.565481] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1768.646222] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1774.049455] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1774.051165] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1774.051762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1774.053151] LustreError: Skipped 1 previous similar message [ 1774.057427] Lustre: Skipped 1 previous similar message [ 1775.378512] Lustre: server umount lustre-OST0000 complete [ 1777.120157] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1777.122878] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1777.128233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1777.130382] Lustre: Skipped 1 previous similar message [ 1779.168610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1779.169087] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1782.813556] Lustre: server umount lustre-MDT0000 complete [ 1784.025100] LustreError: 112368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784343135 with bad export cookie 10890535183213080326 [ 1784.028768] LustreError: 112368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1784.029143] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1784.146058] Lustre: server umount lustre-MDT0001 complete [ 1786.821783] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 22:52:17 (1784343137) [ 1787.888738] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 1793.268703] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 1796.443466] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 1807.761569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1807.872178] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1807.882476] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1807.917439] Lustre: lustre-MDT0000: new disk, initializing [ 1807.940275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1807.946670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1809.332442] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1813.992429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1814.039260] Lustre: 119237:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1814.060739] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1814.062854] Lustre: Skipped 1 previous similar message [ 1814.096272] Lustre: lustre-MDT0001: new disk, initializing [ 1814.142942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1814.147566] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1815.455109] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1819.115652] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1820.915719] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1823.101035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1823.202267] Lustre: lustre-OST0000: new disk, initializing [ 1823.203799] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1823.206390] Lustre: 121202:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1823.230338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1823.232501] Lustre: Skipped 1 previous similar message [ 1825.080621] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1828.347232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1828.351589] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1828.377177] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1828.816319] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1830.208871] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1830.284355] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1831.677405] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1831.758132] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1833.444092] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1833.444114] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1833.449433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1833.457722] Lustre: Skipped 1 previous similar message [ 1833.462282] Lustre: Skipped 1 previous similar message [ 1834.502412] Lustre: server umount lustre-OST0000 complete [ 1838.561981] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1838.562607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1838.567947] Lustre: Skipped 1 previous similar message [ 1841.828294] Lustre: server umount lustre-MDT0000 complete [ 1843.137844] LustreError: 119230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784343194 with bad export cookie 10890535183213081194 [ 1843.141517] LustreError: 119230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1843.142702] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1843.254979] Lustre: server umount lustre-MDT0001 complete [ 1846.140837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1846.327077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1847.749233] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1850.580377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1852.159065] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1854.573967] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1856.351316] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1858.552457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1860.639456] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1863.121199] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1876.013137] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 1881.571205] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1881.575276] LustreError: Skipped 1 previous similar message [ 1881.577684] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1881.584719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1881.587330] Lustre: Skipped 2 previous similar messages [ 1883.023021] Lustre: server umount lustre-OST0000 complete [ 1886.688987] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1886.691213] LustreError: Skipped 1 previous similar message [ 1890.329678] Lustre: server umount lustre-MDT0000 complete [ 1891.500153] LustreError: 123395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784343242 with bad export cookie 10890535183213082048 [ 1891.503109] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1891.503583] LustreError: 123395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1891.606852] Lustre: server umount lustre-MDT0001 complete [ 1895.213126] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 1896.256627] Key type lgssc unregistered [ 1896.399370] LNet: 127060:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1896.401921] LNetError: 127060:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1896.410405] LNet: Removed LNI 192.168.203.149@tcp [ 1896.753513] Key type .llcrypt unregistered [ 1896.754745] Key type ._llcrypt unregistered [ 1902.998480] Key type ._llcrypt registered [ 1902.999989] Key type .llcrypt registered [ 1903.043107] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 1909.435742] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 1909.754684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1909.817026] alg: No test for adler32 (adler32-zlib) [ 1910.660402] Lustre: Lustre: Build Version: 2.17.54_173_gd3f3fcc [ 1910.745892] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 1912.327131] Key type lgssc registered [ 1912.697510] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1923.984650] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1923.992314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1925.082571] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1925.092241] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1925.129534] Lustre: lustre-MDT0000: new disk, initializing [ 1925.150166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1925.156214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1926.492273] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1930.791985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1930.854143] Lustre: 130185:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1930.884115] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1930.886686] Lustre: Skipped 1 previous similar message [ 1930.925833] Lustre: lustre-MDT0001: new disk, initializing [ 1930.959043] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1930.971578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1930.975552] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1932.383891] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1936.010977] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1937.809540] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1940.072927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1940.187107] Lustre: lustre-OST0000: new disk, initializing [ 1940.189738] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1940.192273] Lustre: 132158:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1940.225354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1940.386597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1940.392134] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1940.429849] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1942.096757] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1945.716239] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1946.997517] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1947.061916] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1948.362025] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1948.435451] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1950.692563] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1950.692731] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1950.695320] LustreError: Skipped 1 previous similar message [ 1950.699564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1950.702977] Lustre: Skipped 1 previous similar message [ 1955.148275] Lustre: server umount lustre-OST0000 complete [ 1956.831862] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1956.834766] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1956.838547] Lustre: Skipped 1 previous similar message [ 1956.840376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1960.928127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1960.928654] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1962.460282] Lustre: server umount lustre-MDT0000 complete [ 1963.654266] LustreError: 130177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784343315 with bad export cookie 10646952619539226167 [ 1963.657826] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1963.658422] LustreError: 130177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1963.777149] Lustre: server umount lustre-MDT0001 complete [ 1966.446936] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 22:55:17 (1784343317) [ 1968.595707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1968.785797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1970.090865] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1972.872912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1972.996855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1974.413442] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1976.760608] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1978.563616] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1983.328876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1983.329662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1983.333215] Lustre: Skipped 1 previous similar message [ 1983.335203] Lustre: Skipped 1 previous similar message [ 1985.365574] Lustre: server umount lustre-MDT0000 complete [ 1986.540326] LustreError: 134548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784343338 with bad export cookie 10646952619539227035 [ 1986.544143] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1986.546717] LustreError: 134548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1986.666622] Lustre: server umount lustre-MDT0001 complete [ 1989.236206] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 22:55:40 (1784343340) [ 1991.322914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2006.751458] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2011.871347] LustreError: 136979:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.149@tcp: failed processing log, type 4: rc = -110 [ 2037.471622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2039.553997] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2042.196329] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2042.913287] Lustre: server umount lustre-OST0000 complete [ 2045.542091] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 22:56:36 (1784343396) [ 2047.814575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2047.982238] LustreError: 138422:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2047.991089] LustreError: 138422:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2049.358701] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2052.027023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2053.381120] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2055.655075] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2057.480757] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2059.695252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2059.802620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2059.804695] Lustre: Skipped 2 previous similar messages [ 2061.583973] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2063.957460] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2077.664505] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2077.669890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2077.678957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2082.077865] Lustre: server umount lustre-MDT0000 complete [ 2083.296341] LustreError: 138422:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2083.302780] LustreError: 138422:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2083.315191] LustreError: 141023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784343434 with bad export cookie 10646952619539227686 [ 2083.318492] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2083.321613] LustreError: 141023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2083.438706] Lustre: server umount lustre-MDT0001 complete [ 2094.886258] Lustre: server umount lustre-OST0000 complete [ 2097.793372] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 22:57:28 (1784343448) [ 2099.979583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2100.120464] LustreError: 142095:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2100.142471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2101.394396] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2104.350926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2105.856527] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2108.562755] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2110.262728] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2112.381543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2114.516126] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2116.980904] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2118.467762] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2124.777509] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 2125.768384] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec [ 2128.156636] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2128.269713] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2129.889521] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2129.893605] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2129.895589] LustreError: Skipped 1 previous similar message [ 2129.896312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2129.896316] Lustre: Skipped 3 previous similar messages [ 2129.912265] Lustre: Skipped 3 previous similar messages [ 2131.293437] Lustre: server umount lustre-OST0000 complete [ 2147.295130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2147.359742] Lustre: server umount lustre-MDT0000 complete [ 2149.134456] LustreError: 142078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784343500 with bad export cookie 10646952619539228743 [ 2149.137334] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2149.141169] LustreError: 142078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2149.280882] Lustre: server umount lustre-MDT0001 complete [ 2153.267979] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 22:58:24 (1784343504) [ 2159.633601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2175.199823] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2180.319478] LustreError: 146299:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.149@tcp: failed processing log, type 4: rc = -110 [ 2205.919395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2205.922424] Lustre: Skipped 2 previous similar messages [ 2208.129525] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2211.416055] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2214.713538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2214.912841] LustreError: 146324:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2214.984026] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 2216.763986] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2220.000676] LustreError: 146325:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2220.767524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2222.954586] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2226.685881] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2229.450654] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2231.536970] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2231.641299] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2233.616947] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2233.714951] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2236.385237] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2236.388692] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2236.393843] Lustre: Skipped 2 previous similar messages [ 2236.396138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2236.398462] Lustre: Skipped 7 previous similar messages [ 2240.746581] Lustre: server umount lustre-MDT0000 complete [ 2242.415707] LustreError: 146307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784343593 with bad export cookie 10646952619539229751 [ 2242.422127] LustreError: 146307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2242.422353] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2242.528912] LustreError: 147332:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2242.537137] LustreError: 147332:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2242.590114] Lustre: server umount lustre-MDT0001 complete [ 2250.392419] Lustre: server umount lustre-OST0000 complete [ 2253.722627] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 23:00:04 (1784343604) [ 2256.253095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2271.711656] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2276.831461] LustreError: 150486:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.149@tcp: failed processing log, type 4: rc = -110 [ 2305.207685] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2308.890440] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2312.216236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2312.415507] LustreError: 150511:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2312.481105] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 2314.480423] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2318.593978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2320.739457] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2324.101397] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2326.840975] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2330.063728] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2330.162856] Lustre: lustre-OST0001: new disk, initializing [ 2330.165439] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2330.169282] Lustre: 153370:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2332.946714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2332.953400] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2332.999095] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2333.127815] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2338.224461] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2340.369911] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2340.497232] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2342.591499] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2342.701090] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2344.419834] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2344.423048] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2344.428855] Lustre: Skipped 3 previous similar messages [ 2344.431854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2344.435272] Lustre: Skipped 3 previous similar messages [ 2349.694953] Lustre: server umount lustre-OST0000 complete [ 2353.636322] LustreError: 150502:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2353.655546] LustreError: 150502:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2357.607911] Lustre: server umount lustre-OST0001 complete [ 2365.420380] Lustre: server umount lustre-MDT0000 complete [ 2367.031279] LustreError: 150493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784343718 with bad export cookie 10646952619539230703 [ 2367.033712] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2367.036559] LustreError: 150493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2367.173885] Lustre: server umount lustre-MDT0001 complete [ 2376.876079] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 23:02:07 (1784343727) [ 2377.574476] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 2378.301885] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 2379.025912] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 23:02:09 (1784343729) [ 2381.899846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2382.060762] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2382.076841] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2382.181107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2382.184179] Lustre: Skipped 6 previous similar messages [ 2383.882281] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2387.527324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2387.591577] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2387.604718] Lustre: 156924:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2389.366095] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2392.539573] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2395.007596] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2397.933549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2398.018918] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2400.630193] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2400.810862] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 2403.666043] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2405.379250] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2405.466250] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2407.140328] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2407.233561] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2410.977640] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2410.978283] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2410.980673] LustreError: Skipped 4 previous similar messages [ 2410.981352] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2410.984996] Lustre: Skipped 6 previous similar messages [ 2410.986455] Lustre: Skipped 7 previous similar messages [ 2414.165976] Lustre: server umount lustre-OST0000 complete [ 2428.649322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2430.820897] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2430.826679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2430.826748] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2430.835627] Lustre: Skipped 1 previous similar message [ 2431.014510] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2434.012972] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2436.030658] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2436.140962] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2437.977933] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2438.070849] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2440.555238] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffffa0878aff1000.ost_server_uuid 50 [ 2441.260155] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffffa0878aff1000.ost_server_uuid in FULL state after 0 sec [ 2441.351742] LustreError: 158792:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.49@tcp) returned error from blocking AST (req@00000000d0f7e2cf x1871019520492288 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000763ce028/0x93c192ff5c0cf949 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.49@tcp remote: 0xcd80e81b423167d6 expref: 5 pid: 158792 timeout: 2541 lvb_type: 0 lru_score: 0 lru_type: 0 [ 2441.373058] LustreError: lustre-MDT0000: A client on nid 192.168.203.49@tcp was evicted due to a lock blocking callback time out: rc -107 [ 2441.378588] LustreError: 156919:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.49@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000763ce028/0x93c192ff5c0cf949 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.49@tcp remote: 0xcd80e81b423167d6 expref: 6 pid: 158792 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 2446.307386] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2446.311599] LustreError: Skipped 2 previous similar messages [ 2448.997472] Lustre: server umount lustre-OST0000 complete [ 2456.817366] Lustre: server umount lustre-MDT0000 complete [ 2458.540284] LustreError: 157665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784343810 with bad export cookie 10646952619539231900 [ 2458.542983] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2458.546070] LustreError: 157665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2464.777112] Lustre: server umount lustre-MDT0001 complete [ 2469.720312] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 2471.080545] Key type lgssc unregistered [ 2471.244239] LNet: 163029:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2471.249313] LNetError: 163029:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2471.258336] LNet: Removed LNI 192.168.203.149@tcp [ 2471.636121] Key type .llcrypt unregistered [ 2471.638867] Key type ._llcrypt unregistered [ 2483.369522] Key type ._llcrypt registered [ 2483.371705] Key type .llcrypt registered [ 2483.431434] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2483.971538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2483.982873] alg: No test for adler32 (adler32-zlib) [ 2484.891874] Lustre: Lustre: Build Version: 2.17.54_173_gd3f3fcc [ 2485.031356] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 2486.647883] Key type lgssc registered [ 2487.189092] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2491.601253] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2491.635234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2492.839476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2494.727671] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2498.823896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2499.026441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2500.818796] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2504.396546] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2507.237165] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2510.357988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2510.537971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2513.117386] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2516.544913] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2517.734478] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 2528.992562] Lustre: Failing over lustre-MDT0000 [ 2529.155180] Lustre: server umount lustre-MDT0000 complete [ 2529.251126] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2529.254345] LustreError: 164418:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2529.260968] Lustre: Skipped 2 previous similar messages [ 2529.272186] LustreError: 164418:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2534.370046] LustreError: 164417:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2534.379267] LustreError: 164417:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2539.489346] LustreError: 164417:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2539.497782] LustreError: 164417:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2540.348392] LustreError: 167013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.49@tcp arrived at 1784343891 with bad export cookie 1567497210540131647 [ 2540.358114] LustreError: 167013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2543.752526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2543.836138] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2543.962658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2543.995817] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2545.922956] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2546.877703] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2546.881565] Lustre: lustre-MDT0000: Denying connection for new client 41327c5e-9714-4cf8-8c7d-a82755b349e0 (at 192.168.203.49@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2549.222125] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2554.338864] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2554.349046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2554.352113] Lustre: Skipped 1 previous similar message [ 2559.458518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2559.461229] Lustre: Skipped 1 previous similar message [ 2564.578736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2564.583433] Lustre: Skipped 1 previous similar message [ 2568.159272] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2568.223393] Lustre: server umount lustre-OST0000 complete [ 2569.858811] LustreError: 168160:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2569.862821] Lustre: 167513:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2569.867306] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2569.876100] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2569.881837] LustreError: 167513:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000dde98261 x1871020122216320/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2569.889814] LustreError: 167513:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 2569.895918] LustreError: 167513:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 2569.902488] Lustre: lustre-MDT0000: Recovery over after 0:23, of 2 clients 0 recovered and 2 were evicted. [ 2570.026615] Lustre: server umount lustre-MDT0000 complete [ 2571.568430] LustreError: 164398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784343923 with bad export cookie 1567497210540131983 [ 2571.573566] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2571.574633] LustreError: 164398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2571.579317] LustreError: 163728:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 2577.796831] Lustre: server umount lustre-MDT0001 complete [ 2582.124228] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 2583.290977] Key type lgssc unregistered [ 2583.435523] LNet: 168912:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2583.441310] LNetError: 168912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2583.453670] LNet: Removed LNI 192.168.203.149@tcp [ 2583.809280] Key type .llcrypt unregistered [ 2583.811048] Key type ._llcrypt unregistered [ 2594.379518] Key type ._llcrypt registered [ 2594.381373] Key type .llcrypt registered [ 2594.433903] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2594.900292] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2594.913887] alg: No test for adler32 (adler32-zlib) [ 2595.810653] Lustre: Lustre: Build Version: 2.17.54_173_gd3f3fcc [ 2595.921261] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 2597.519171] Key type lgssc registered [ 2598.014893] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2601.310589] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2601.320088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2602.517331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2604.137703] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2607.434790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2607.589359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2609.224753] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2612.191722] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2614.343809] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2616.977653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2617.148174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2619.445093] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2622.294918] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2627.366366] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 2632.674057] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2632.674504] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2632.674960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2632.678710] Lustre: Skipped 1 previous similar message [ 2635.992398] Lustre: server umount lustre-OST0000 complete [ 2637.792399] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2637.793626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2637.799146] Lustre: Skipped 1 previous similar message [ 2637.804872] Lustre: Skipped 2 previous similar messages [ 2642.912308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2642.915678] Lustre: Skipped 1 previous similar message [ 2643.548749] Lustre: server umount lustre-MDT0000 complete [ 2644.953226] LustreError: 170282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784343996 with bad export cookie 7425714172880764554 [ 2644.956307] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2644.958610] LustreError: 170282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2645.090398] Lustre: server umount lustre-MDT0001 complete [ 2649.169317] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 2650.292125] Key type lgssc unregistered [ 2650.427534] LNet: 173832:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2650.431462] LNetError: 173832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2650.439462] LNet: Removed LNI 192.168.203.149@tcp [ 2650.818982] Key type .llcrypt unregistered [ 2650.821296] Key type ._llcrypt unregistered [ 2663.862710] Key type ._llcrypt registered [ 2663.863960] Key type .llcrypt registered [ 2663.906263] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2664.430470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2664.463770] alg: No test for adler32 (adler32-zlib) [ 2665.326691] Lustre: Lustre: Build Version: 2.17.54_173_gd3f3fcc [ 2665.434819] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 2667.031146] Key type lgssc registered [ 2667.458985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2670.498481] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2670.508543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2671.679628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2673.275517] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2676.645927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2676.807980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2678.562781] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2681.608983] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2683.926883] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2686.537582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2686.671784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2688.860750] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2691.628397] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2697.959057] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 2708.449185] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2708.450148] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2708.453784] LustreError: Skipped 1 previous similar message [ 2708.459282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2708.463032] Lustre: Skipped 1 previous similar message [ 2710.711557] Lustre: server umount lustre-OST0000 complete [ 2712.544073] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2712.547565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2712.556464] Lustre: Skipped 1 previous similar message [ 2712.559936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2713.568704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2713.568990] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2718.507253] Lustre: server umount lustre-MDT0000 complete [ 2718.688917] LustreError: 175505:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2718.698269] LustreError: 175505:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2720.140278] LustreError: 177353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784344071 with bad export cookie 14896456539662907705 [ 2720.142904] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2720.146756] LustreError: 177353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2720.288414] Lustre: server umount lustre-MDT0001 complete [ 2724.551985] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 2725.842945] Key type lgssc unregistered [ 2725.999159] LNet: 179063:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2726.004464] LNetError: 179063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2726.014598] LNet: Removed LNI 192.168.203.149@tcp [ 2726.397807] Key type .llcrypt unregistered [ 2726.400645] Key type ._llcrypt unregistered [ 2736.267192] Key type ._llcrypt registered [ 2736.269022] Key type .llcrypt registered [ 2736.312374] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2736.772200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2736.783119] alg: No test for adler32 (adler32-zlib) [ 2737.656289] Lustre: Lustre: Build Version: 2.17.54_173_gd3f3fcc [ 2737.763272] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 2739.353506] Key type lgssc registered [ 2739.802637] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2744.079292] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2744.087603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2745.248697] Lustre: *** cfs_fail_loc=135, val=0*** [ 2745.292573] LustreError: 180564:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2745.296087] LustreError: 180564:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.149@tcp: cfg command failed: rc = -2 [ 2745.301293] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2745.301293] [ 2745.306466] LustreError: MGC192.168.203.149@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2745.313639] LustreError: 180525:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2745.319043] LustreError: 180525:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2745.323550] LustreError: 180525:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2745.327616] LustreError: 180525:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2745.362859] Lustre: server umount lustre-MDT0000 complete [ 2745.365139] LustreError: 180525:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2747.868788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2748.043649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2749.520788] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2752.665681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2752.810569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2754.306064] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2757.060374] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2759.124661] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2762.977699] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2762.978903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2762.983272] Lustre: Skipped 1 previous similar message [ 2766.047415] Lustre: server umount lustre-MDT0000 complete [ 2767.551152] LustreError: 182499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784344119 with bad export cookie 13548059164791233812 [ 2767.554450] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2767.556549] LustreError: 182499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2767.699178] Lustre: server umount lustre-MDT0001 complete [ 2771.853735] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 2773.014688] Key type lgssc unregistered [ 2773.153423] LNet: 183252:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2773.156878] LNetError: 183252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2773.164309] LNet: Removed LNI 192.168.203.149@tcp [ 2773.493130] Key type .llcrypt unregistered [ 2773.495172] Key type ._llcrypt unregistered [ 2781.572930] Key type ._llcrypt registered [ 2781.575281] Key type .llcrypt registered [ 2781.621183] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 2781.747492] Key type .llcrypt unregistered [ 2781.749319] Key type ._llcrypt unregistered [ 2789.383204] Key type ._llcrypt registered [ 2789.384750] Key type .llcrypt registered [ 2789.431596] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2789.896584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2789.940720] alg: No test for adler32 (adler32-zlib) [ 2790.825264] Lustre: Lustre: Build Version: 2.17.54_173_gd3f3fcc [ 2790.928799] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 2792.527177] Key type lgssc registered [ 2792.960371] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2795.967325] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2795.977800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2811.423383] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2816.543293] LustreError: 185360:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.149@tcp: failed processing log, type 4: rc = -110 [ 2843.167420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2845.571386] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2848.627769] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2851.280079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2851.516638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2851.546415] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2853.088202] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2856.474827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2856.633357] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2858.197405] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2860.943216] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2863.140396] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2864.330797] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 2868.193670] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2868.195193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2868.199680] Lustre: Skipped 1 previous similar message [ 2873.312706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2873.315716] Lustre: Skipped 2 previous similar messages [ 2874.198615] Lustre: server umount lustre-OST0000 complete [ 2877.407997] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2877.412108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2877.417967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2878.433707] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2881.686394] Lustre: server umount lustre-MDT0000 complete [ 2883.016148] LustreError: 185366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784344234 with bad export cookie 13077740361686100145 [ 2883.019387] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2883.021381] LustreError: 185366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2883.136649] Lustre: server umount lustre-MDT0001 complete [ 2887.126997] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 2888.294147] Key type lgssc unregistered [ 2888.445466] LNet: 189220:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2888.449053] LNetError: 189220:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2888.460666] LNet: Removed LNI 192.168.203.149@tcp [ 2888.820503] Key type .llcrypt unregistered [ 2888.821756] Key type ._llcrypt unregistered [ 2899.110932] Key type ._llcrypt registered [ 2899.112500] Key type .llcrypt registered [ 2899.155545] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2899.485538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2899.497213] alg: No test for adler32 (adler32-zlib) [ 2900.365768] Lustre: Lustre: Build Version: 2.17.54_173_gd3f3fcc [ 2900.462740] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 2902.055171] Key type lgssc registered [ 2902.509297] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2905.624287] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2905.633866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2906.812814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2908.466795] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2911.905173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2912.073230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2913.644183] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2916.594792] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2918.734127] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2921.353936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2921.490071] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2923.610766] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2926.354953] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2931.750708] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 2933.042727] Lustre: Failing over lustre-MDT0000 [ 2933.140690] Lustre: server umount lustre-MDT0000 complete [ 2936.801062] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2936.802437] LustreError: 191931:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2936.805495] Lustre: Skipped 2 previous similar messages [ 2936.813829] LustreError: 191931:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2941.920628] LustreError: 190607:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2941.927727] LustreError: 190607:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2946.326103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2946.402787] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2946.507156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2946.535862] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2947.254914] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2947.996574] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2949.664084] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 2951.657528] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2951.664993] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2951.682534] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 2957.049102] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 2968.543883] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2968.548550] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2968.554504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2972.132454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2972.133230] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2972.134261] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2973.583363] Lustre: server umount lustre-OST0000 complete [ 2977.252614] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2977.258881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2977.260651] Lustre: Skipped 1 previous similar message [ 2977.268061] Lustre: Skipped 1 previous similar message [ 2981.409991] Lustre: server umount lustre-MDT0000 complete [ 2982.370294] LustreError: 190604:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2982.378480] LustreError: 190604:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2984.309627] LustreError: 194525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784344335 with bad export cookie 16243661907223120142 [ 2984.314967] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2984.317921] LustreError: 194525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2984.980716] Lustre: server umount lustre-MDT0001 complete [ 2993.361427] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 2995.175582] Key type lgssc unregistered [ 2995.637482] LNet: 195583:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2995.643299] LNetError: 195583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2995.656149] LNet: Removed LNI 192.168.203.149@tcp [ 2996.209570] Key type .llcrypt unregistered [ 2996.211271] Key type ._llcrypt unregistered [ 3021.088938] Key type ._llcrypt registered [ 3021.098037] Key type .llcrypt registered [ 3021.210966] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 3022.587497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3022.791240] alg: No test for adler32 (adler32-zlib) [ 3023.937699] Lustre: Lustre: Build Version: 2.17.54_173_gd3f3fcc [ 3023.962658] hrtimer: interrupt took 6649689 ns [ 3024.245828] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 3025.919153] Key type lgssc registered [ 3028.201612] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3038.459172] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3038.480493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3039.842500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3044.373522] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3050.465659] LustreError: 196996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3050.482039] LustreError: 196996:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3051.924514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3052.209181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3055.371645] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3059.847362] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3063.125948] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3067.139759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3067.388673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3070.657212] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3074.522964] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3084.775866] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 3086.983510] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3089.657770] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 3100.707256] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3107.296845] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3107.302800] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3107.318805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3110.372341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3110.374390] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3110.387178] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3113.325739] Lustre: server umount lustre-OST0000 complete [ 3119.084121] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3119.092158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3119.110052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3119.118764] Lustre: Skipped 1 previous similar message [ 3122.330196] Lustre: server umount lustre-MDT0000 complete [ 3124.898827] LustreError: 196977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784344476 with bad export cookie 6488391242665735766 [ 3124.904413] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3124.914883] LustreError: 196977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3125.164805] Lustre: server umount lustre-MDT0001 complete [ 3131.415669] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 3133.182910] Key type lgssc unregistered [ 3133.373641] LNet: 200863:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3133.378371] LNetError: 200863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3133.397114] LNet: Removed LNI 192.168.203.149@tcp [ 3133.914987] Key type .llcrypt unregistered [ 3133.919666] Key type ._llcrypt unregistered [ 3145.197177] Key type ._llcrypt registered [ 3145.198787] Key type .llcrypt registered [ 3145.271548] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 3145.700202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3145.738272] alg: No test for adler32 (adler32-zlib) [ 3146.695654] Lustre: Lustre: Build Version: 2.17.54_173_gd3f3fcc [ 3146.842338] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 3148.447266] Key type lgssc registered [ 3149.096749] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3154.007734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3154.031753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3155.311990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3157.687225] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3163.517127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3163.818830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3166.804374] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3171.557317] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3175.397652] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3179.589759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3179.837990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3180.846243] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 3184.432224] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3190.124523] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3195.054180] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 3200.297841] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3209.020776] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 3218.403777] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 3225.059198] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3225.063103] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3225.064067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3225.083391] Lustre: Skipped 1 previous similar message [ 3230.180428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3230.186547] Lustre: Skipped 1 previous similar message [ 3230.335473] Lustre: server umount lustre-OST0000 complete [ 3235.296406] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3235.299455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3235.307465] Lustre: Skipped 1 previous similar message [ 3235.309945] Lustre: Skipped 1 previous similar message [ 3240.418447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3240.424692] Lustre: Skipped 2 previous similar messages [ 3245.538165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3245.542496] Lustre: Skipped 1 previous similar message [ 3247.071190] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3247.189970] Lustre: server umount lustre-MDT0000 complete [ 3249.702103] LustreError: 202841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784344601 with bad export cookie 12481255878027813030 [ 3249.703957] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3249.710330] LustreError: 202841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3249.974203] Lustre: server umount lustre-MDT0001 complete [ 3257.378267] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 3259.354857] Key type lgssc unregistered [ 3259.596062] LNet: 207173:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3259.605869] LNetError: 207173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3259.623947] LNet: Removed LNI 192.168.203.149@tcp [ 3260.255502] Key type .llcrypt unregistered [ 3260.258314] Key type ._llcrypt unregistered [ 3276.126944] Key type ._llcrypt registered [ 3276.129153] Key type .llcrypt registered [ 3276.215992] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 3276.811230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3276.858898] alg: No test for adler32 (adler32-zlib) [ 3277.834668] Lustre: Lustre: Build Version: 2.17.54_173_gd3f3fcc [ 3278.001722] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 3279.623227] Key type lgssc registered [ 3280.492579] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3285.756941] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3285.769149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3287.021669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3289.957677] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3295.949152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3296.172605] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3299.200647] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3304.294793] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3308.154820] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3312.958769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3313.269054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3317.002328] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3321.869730] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3331.687284] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 3342.309402] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3342.311686] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3342.316956] LustreError: Skipped 1 previous similar message [ 3342.333431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3342.344655] Lustre: Skipped 1 previous similar message [ 3346.023669] Lustre: server umount lustre-OST0000 complete [ 3352.546481] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3352.551312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3352.552560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3352.557414] Lustre: Skipped 2 previous similar messages [ 3354.612184] Lustre: server umount lustre-MDT0000 complete [ 3357.064234] LustreError: 211513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784344708 with bad export cookie 16109646120780156986 [ 3357.069045] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3357.076082] LustreError: 211513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3357.360449] Lustre: server umount lustre-MDT0001 complete [ 3364.234194] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 3366.337161] Key type lgssc unregistered [ 3366.563567] LNet: 212264:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3366.570847] LNetError: 212264:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3366.584673] LNet: Removed LNI 192.168.203.149@tcp [ 3367.170583] Key type .llcrypt unregistered [ 3367.175083] Key type ._llcrypt unregistered [ 3381.118726] Key type ._llcrypt registered [ 3381.120775] Key type .llcrypt registered [ 3381.210306] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 3381.875606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3381.885079] alg: No test for adler32 (adler32-zlib) [ 3382.932691] Lustre: Lustre: Build Version: 2.17.54_173_gd3f3fcc [ 3383.147717] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 3384.791965] Key type lgssc registered [ 3385.774739] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3391.222422] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3391.235674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3392.511632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3395.335486] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3400.648932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3400.889961] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3403.485589] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3408.522263] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3412.483867] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3417.356992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3417.589314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3421.896729] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3426.893107] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3437.052144] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 3447.272810] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3447.292481] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3447.295395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3447.313890] Lustre: Skipped 1 previous similar message [ 3452.385439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3452.395090] Lustre: Skipped 1 previous similar message [ 3452.609824] Lustre: server umount lustre-OST0000 complete [ 3457.509757] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3457.517366] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3457.517817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3457.525537] Lustre: Skipped 2 previous similar messages [ 3461.376120] Lustre: server umount lustre-MDT0000 complete [ 3462.627296] LustreError: 216079:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3462.644168] LustreError: 216079:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3463.729288] LustreError: 213490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784344815 with bad export cookie 155986598247956135 [ 3463.733953] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3463.740214] LustreError: 213490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3463.966223] Lustre: server umount lustre-MDT0001 complete [ 3471.248952] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 3473.278564] Key type lgssc unregistered [ 3473.524025] LNet: 217210:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3473.532582] LNetError: 217210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3473.544980] LNet: Removed LNI 192.168.203.149@tcp [ 3474.164111] Key type .llcrypt unregistered [ 3474.169300] Key type ._llcrypt unregistered [ 3486.428898] Key type ._llcrypt registered [ 3486.430842] Key type .llcrypt registered [ 3486.507874] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 3487.116208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3487.151068] alg: No test for adler32 (adler32-zlib) [ 3488.147207] Lustre: Lustre: Build Version: 2.17.54_173_gd3f3fcc [ 3488.309361] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 3489.935830] Key type lgssc registered [ 3490.665125] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3495.452223] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3495.462899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3496.747316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3499.291224] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3504.979598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3505.199619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3508.137784] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3512.994527] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3516.655664] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3520.882328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3521.212804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3522.228300] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 3524.751565] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3529.224954] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3534.967687] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3535.021213] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3535.190937] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3538.856643] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3543.715480] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3544.834658] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3555.993741] Lustre: Permanently deactivating lustre-OST0001 [ 3556.002732] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3565.598914] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3572.412866] Lustre: Permanently reactivating lustre-OST0001 [ 3572.420582] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 3572.425824] Lustre: Skipped 2 previous similar messages [ 3575.458250] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3575.466468] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3575.470964] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3575.481759] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3575.518768] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3585.002883] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3585.005557] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3585.012454] Lustre: Skipped 2 previous similar messages [ 3590.113624] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3590.117568] Lustre: Skipped 2 previous similar messages [ 3590.589639] Lustre: server umount lustre-OST0001 complete [ 3594.208722] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3594.218197] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3594.226797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3595.235408] LustreError: 220273:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3595.243114] LustreError: 220273:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3598.974490] Lustre: server umount lustre-OST0000 complete [ 3602.400745] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3602.406833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3602.412584] Lustre: Skipped 1 previous similar message [ 3602.416493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3602.419771] Lustre: Skipped 2 previous similar messages [ 3607.436946] Lustre: server umount lustre-MDT0000 complete [ 3609.749920] LustreError: 221290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784344961 with bad export cookie 9853551540781073731 [ 3609.752061] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3609.756574] LustreError: 221290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3609.944644] Lustre: server umount lustre-MDT0001 complete [ 3615.674635] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 3617.165469] Key type lgssc unregistered [ 3617.333816] LNet: 223564:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3617.341268] LNetError: 223564:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3617.354686] LNet: Removed LNI 192.168.203.149@tcp [ 3617.824916] Key type .llcrypt unregistered [ 3617.827464] Key type ._llcrypt unregistered [ 3642.043435] Key type ._llcrypt registered [ 3642.047794] Key type .llcrypt registered [ 3642.127993] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 3642.717215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3642.798863] alg: No test for adler32 (adler32-zlib) [ 3643.769459] Lustre: Lustre: Build Version: 2.17.54_173_gd3f3fcc [ 3643.939702] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 3645.607247] Key type lgssc registered [ 3646.418396] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3650.972081] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3650.982442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3652.152750] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3652.171986] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3652.259950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3654.889447] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3659.847753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3659.904915] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3659.924341] Lustre: 225745:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3660.079340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3662.477457] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3666.726490] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3669.876418] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3673.748612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3673.860469] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3673.980978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3677.245051] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3681.575137] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3681.854708] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 3684.601507] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 3689.399814] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 3698.538363] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 3706.561568] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3721.323972] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3721.331077] Lustre: Skipped 1 previous similar message [ 3745.604357] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 3745.610793] Lustre: Skipped 2 previous similar messages [ 3785.019425] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3785.025330] Lustre: Skipped 4 previous similar messages [ 3852.946377] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 3852.957452] Lustre: Skipped 7 previous similar messages [ 3864.245992] Lustre: server umount lustre-OST0000 complete [ 3869.667698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3869.669442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3869.678285] Lustre: Skipped 1 previous similar message [ 3869.685845] Lustre: Skipped 1 previous similar message [ 3874.787691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3874.797533] Lustre: Skipped 1 previous similar message [ 3877.935028] Lustre: server umount lustre-MDT0000 complete [ 3880.929831] Lustre: 225070:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784345216/real 1784345216] req@0000000053331fd8 x1871021337660288/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1784345232 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3880.968263] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3881.720477] LustreError: 225737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784345233 with bad export cookie 6928652352280350086 [ 3881.727436] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3881.730126] LustreError: 225737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3882.051123] Lustre: server umount lustre-MDT0001 complete [ 3891.864211] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 3894.205610] Key type lgssc unregistered [ 3894.443679] LNet: 230559:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3894.453580] LNetError: 230559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3894.474087] LNet: Removed LNI 192.168.203.149@tcp [ 3895.052851] Key type .llcrypt unregistered [ 3895.058573] Key type ._llcrypt unregistered [ 3913.906682] Key type ._llcrypt registered [ 3913.915021] Key type .llcrypt registered [ 3914.002631] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 3914.982165] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3915.011286] alg: No test for adler32 (adler32-zlib) [ 3916.021474] Lustre: Lustre: Build Version: 2.17.54_173_gd3f3fcc [ 3916.247433] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 3917.889292] Key type lgssc registered [ 3918.882641] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3925.287510] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3925.315054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3926.781254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3930.096294] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3936.749332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3937.108360] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3940.510608] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3946.516360] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3951.016323] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3956.385511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3956.703981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3961.358238] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3966.693609] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3973.105597] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 3992.544855] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3992.550245] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3992.569843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3993.569112] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3993.572168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3993.576848] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3993.587864] Lustre: Skipped 1 previous similar message [ 3994.987907] Lustre: server umount lustre-OST0000 complete [ 3998.689590] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3998.690391] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3998.691896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3998.719431] Lustre: Skipped 1 previous similar message [ 4003.808619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4003.814345] Lustre: Skipped 2 previous similar messages [ 4004.247903] Lustre: server umount lustre-MDT0000 complete [ 4007.211769] LustreError: 231933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784345358 with bad export cookie 17004025102534897659 [ 4007.213428] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4007.224029] LustreError: 231933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4007.548086] Lustre: server umount lustre-MDT0001 complete [ 4015.889351] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 4018.088741] Key type lgssc unregistered [ 4018.331994] LNet: 235640:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4018.342384] LNetError: 235640:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4018.361793] LNet: Removed LNI 192.168.203.149@tcp [ 4018.999645] Key type .llcrypt unregistered [ 4019.002206] Key type ._llcrypt unregistered [ 4036.764583] Key type ._llcrypt registered [ 4036.766839] Key type .llcrypt registered [ 4036.871215] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 4037.746161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4037.894450] alg: No test for adler32 (adler32-zlib) [ 4038.903477] Lustre: Lustre: Build Version: 2.17.54_173_gd3f3fcc [ 4039.100865] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 4040.759181] Key type lgssc registered [ 4041.786645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4308.865376] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 4311.033532] Key type lgssc unregistered [ 4311.286620] LNet: 237641:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4311.293592] LNetError: 237641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4311.312121] LNet: Removed LNI 192.168.203.149@tcp [ 4311.856428] Key type .llcrypt unregistered [ 4311.862265] Key type ._llcrypt unregistered [ 4332.387225] Key type ._llcrypt registered [ 4332.390339] Key type .llcrypt registered [ 4332.473712] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 4344.372924] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 4344.988412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4345.112237] alg: No test for adler32 (adler32-zlib) [ 4346.120861] Lustre: Lustre: Build Version: 2.17.54_173_gd3f3fcc [ 4346.284507] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 4347.919458] Key type lgssc registered [ 4348.772201] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4373.268621] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 4245 sec ========== 23:35:23 (1784345723) [ 4374.813051] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:35:24 (1784345724) === [ 4377.526967] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:35:27 (1784345727) === [ 4408.967437] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 4411.714321] Key type lgssc unregistered [ 4411.974897] LNet: 242854:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4411.981256] LNetError: 242854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4411.997562] LNet: Removed LNI 192.168.203.149@tcp [ 4412.811656] Key type .llcrypt unregistered [ 4412.816339] Key type ._llcrypt unregistered