[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 550518505 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003129] x2apic enabled [ 0.004003] Switched APIC routing to physical x2apic. [ 0.005006] kvm-guest: setup PV IPIs [ 0.006950] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007011] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008004] pid_max: default: 32768 minimum: 301 [ 0.009085] LSM: Security Framework initializing [ 0.010031] Yama: becoming mindful. [ 0.011023] SELinux: Initializing. [ 0.011913] *** VALIDATE selinux *** [ 0.018723] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.022531] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.023125] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024093] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025069] *** VALIDATE tmpfs *** [ 0.026383] *** VALIDATE proc *** [ 0.028046] *** VALIDATE cgroup *** [ 0.029005] *** VALIDATE cgroup2 *** [ 0.030226] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.031175] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.032003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.033022] Spectre V2 : User space: Vulnerable [ 0.034004] Speculative Store Bypass: Vulnerable [ 0.037637] debug: unmapping init [mem 0xffffffffba859000-0xffffffffba860fff] [ 0.039240] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.040680] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.041015] ... version: 2 [ 0.042007] ... bit width: 48 [ 0.043008] ... generic registers: 4 [ 0.044007] ... value mask: 0000ffffffffffff [ 0.045006] ... max period: 00007fffffffffff [ 0.046006] ... fixed-purpose events: 3 [ 0.047007] ... event mask: 000000070000000f [ 0.048250] rcu: Hierarchical SRCU implementation. [ 0.050504] smp: Bringing up secondary CPUs ... [ 0.051682] x86: Booting SMP configuration: [ 0.052014] .... node #0, CPUs: #1 #2 #3 [ 0.059110] smp: Brought up 1 node, 4 CPUs [ 0.061009] smpboot: Max logical packages: 1 [ 0.062012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.105557] node 0 deferred pages initialised in 42ms [ 0.110008] devtmpfs: initialized [ 0.111088] x86/mm: Memory block size: 128MB [ 0.114199] gcov: version magic: 0x41383552 [ 0.116197] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.117060] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.118211] pinctrl core: initialized pinctrl subsystem [ 0.119137] [ 0.119798] ************************************************************* [ 0.120006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.121008] ** ** [ 0.122007] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.123005] ** ** [ 0.124005] ** This means that this kernel is built to expose internal ** [ 0.125006] ** IOMMU data structures, which may compromise security on ** [ 0.126005] ** your system. ** [ 0.127005] ** ** [ 0.128005] ** If you see this message and you are not debugging the ** [ 0.129005] ** kernel, report this immediately to your vendor! ** [ 0.130005] ** ** [ 0.131004] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.132005] ************************************************************* [ 0.133760] NET: Registered protocol family 16 [ 0.137886] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.145088] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.153174] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.161011] cpuidle: using governor menu [ 0.163267] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.165327] PCI: Using configuration type 1 for base access [ 0.166000] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.174320] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.175021] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.176383] cryptd: max_cpu_qlen set to 1000 [ 0.177000] ACPI: Added _OSI(Module Device) [ 0.177022] ACPI: Added _OSI(Processor Device) [ 0.178008] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.179000] ACPI: Added _OSI(Processor Aggregator Device) [ 0.184577] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.192496] ACPI: Interpreter enabled [ 0.194041] ACPI: PM: (supports S0 S3 S4 S5) [ 0.195005] ACPI: Using IOAPIC for interrupt routing [ 0.196048] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.199322] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.207657] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.210020] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.212048] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.216042] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.221434] acpiphp: Slot [2] registered [ 0.222070] acpiphp: Slot [5] registered [ 0.224071] acpiphp: Slot [6] registered [ 0.225064] acpiphp: Slot [7] registered [ 0.226174] acpiphp: Slot [8] registered [ 0.228062] acpiphp: Slot [9] registered [ 0.229075] acpiphp: Slot [10] registered [ 0.230062] acpiphp: Slot [3] registered [ 0.232060] acpiphp: Slot [4] registered [ 0.233072] acpiphp: Slot [11] registered [ 0.234053] acpiphp: Slot [12] registered [ 0.235198] acpiphp: Slot [13] registered [ 0.237056] acpiphp: Slot [14] registered [ 0.238056] acpiphp: Slot [15] registered [ 0.239060] acpiphp: Slot [16] registered [ 0.241068] acpiphp: Slot [17] registered [ 0.242060] acpiphp: Slot [18] registered [ 0.244061] acpiphp: Slot [19] registered [ 0.245055] acpiphp: Slot [20] registered [ 0.247099] acpiphp: Slot [21] registered [ 0.248057] acpiphp: Slot [22] registered [ 0.250065] acpiphp: Slot [23] registered [ 0.251061] acpiphp: Slot [24] registered [ 0.253063] acpiphp: Slot [25] registered [ 0.254061] acpiphp: Slot [26] registered [ 0.256130] acpiphp: Slot [27] registered [ 0.257063] acpiphp: Slot [28] registered [ 0.259064] acpiphp: Slot [29] registered [ 0.261096] acpiphp: Slot [30] registered [ 0.262079] acpiphp: Slot [31] registered [ 0.264046] PCI host bridge to bus 0000:00 [ 0.266044] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.268012] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.271009] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.273010] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.276087] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.279016] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.281157] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.283970] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.287474] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.298813] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.304061] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.306009] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.308009] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.311008] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.313412] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.317635] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.322029] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.327412] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.332011] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.350011] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.356009] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.363174] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.376013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.388017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.427018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.456054] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.471014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.482018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.497014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.508537] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.517011] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.528015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.567015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.589103] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.608015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.621013] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.639020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.653145] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.663016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.669017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.686020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.693730] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.705020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.711016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.743016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.756123] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.758427] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.760313] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.762404] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.764266] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.769095] iommu: Default domain type: Passthrough [ 0.770581] SCSI subsystem initialized [ 0.772157] ACPI: bus type USB registered [ 0.774126] usbcore: registered new interface driver usbfs [ 0.775053] usbcore: registered new interface driver hub [ 0.777143] usbcore: registered new device driver usb [ 0.779126] pps_core: LinuxPPS API ver. 1 registered [ 0.780006] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.783056] PTP clock support registered [ 0.785061] EDAC MC: Ver: 3.0.0 [ 0.786432] PCI: Using ACPI for IRQ routing [ 0.787745] NetLabel: Initializing [ 0.789010] NetLabel: domain hash size = 128 [ 0.790006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.792119] NetLabel: unlabeled traffic allowed by default [ 0.793352] vgaarb: loaded [ 0.795607] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.797007] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.803605] clocksource: Switched to clocksource kvm-clock [ 0.929284] VFS: Disk quotas dquot_6.6.0 [ 0.930815] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.933027] *** VALIDATE ramfs *** [ 0.934122] *** VALIDATE hugetlbfs *** [ 0.935552] pnp: PnP ACPI init [ 0.937812] pnp: PnP ACPI: found 6 devices [ 0.953572] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.956588] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.958481] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.960435] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.962454] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.964936] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.967974] NET: Registered protocol family 2 [ 0.970684] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.976145] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.979704] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.985112] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.988433] TCP: Hash tables configured (established 65536 bind 65536) [ 0.991265] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.994670] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.997967] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.000971] NET: Registered protocol family 1 [ 1.003601] RPC: Registered named UNIX socket transport module. [ 1.005906] RPC: Registered udp transport module. [ 1.007585] RPC: Registered tcp transport module. [ 1.008906] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.011476] NET: Registered protocol family 44 [ 1.014601] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.016761] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.023248] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.025777] PCI: CLS 0 bytes, default 64 [ 1.029142] Unpacking initramfs... [ 2.553945] debug: unmapping init [mem 0xffff8cf3fcc54000-0xffff8cf3fffbffff] [ 2.560666] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.562393] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.564829] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.060022] Initialise system trusted keyrings [ 3.063317] Key type blacklist registered [ 3.066556] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.076861] zbud: loaded [ 3.079733] *** VALIDATE nfs *** [ 3.080749] *** VALIDATE nfs4 *** [ 3.082470] pstore: using deflate compression [ 3.086084] Platform Keyring initialized [ 3.198564] NET: Registered protocol family 38 [ 3.199974] Key type asymmetric registered [ 3.201187] Asymmetric key parser 'x509' registered [ 3.202708] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.206503] io scheduler mq-deadline registered [ 3.207663] io scheduler kyber registered [ 3.208908] io scheduler bfq registered [ 3.211441] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.216172] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.219192] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.221589] ACPI: Power Button [PWRF] [ 3.226305] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.234371] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.250759] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.260144] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.303374] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.331133] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.361298] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.366739] Non-volatile memory driver v1.3 [ 3.368457] Linux agpgart interface v0.103 [ 3.400577] virtio_blk virtio1: [vda] 145912 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.402961] vda: detected capacity change from 0 to 74706944 [ 3.436463] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.439456] vdb: detected capacity change from 0 to 1073741824 [ 3.464723] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.467966] vdc: detected capacity change from 0 to 2621440000 [ 3.521559] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.528457] vdd: detected capacity change from 0 to 2621440000 [ 3.566252] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.569430] vde: detected capacity change from 0 to 4294967296 [ 3.610559] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.617428] vdf: detected capacity change from 0 to 4294967296 [ 3.641454] libphy: Fixed MDIO Bus: probed [ 3.656684] usbcore: registered new interface driver usbserial_generic [ 3.659024] usbserial: USB Serial support registered for generic [ 3.663951] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.673476] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.677430] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.681260] mousedev: PS/2 mouse device common for all mice [ 3.686172] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.686512] rtc_cmos 00:05: RTC can wake from S4 [ 3.699421] rtc_cmos 00:05: registered as rtc0 [ 3.709319] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.711981] intel_pstate: CPU model not supported [ 3.733954] hid: raw HID events driver (C) Jiri Kosina [ 3.736508] usbcore: registered new interface driver usbhid [ 3.738426] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.743430] usbhid: USB HID core driver [ 3.748044] drop_monitor: Initializing network drop monitor service [ 3.757878] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.765252] Initializing XFRM netlink socket [ 3.766307] NET: Registered protocol family 10 [ 3.768111] Segment Routing with IPv6 [ 3.782126] NET: Registered protocol family 17 [ 3.786408] mpls_gso: MPLS GSO support [ 3.801188] RAS: Correctable Errors collector initialized. [ 3.804132] AVX version of gcm_enc/dec engaged. [ 3.808619] AES CTR mode by8 optimization enabled [ 3.918831] sched_clock: Marking stable (3918757970, 0)->(4895517348, -976759378) [ 3.922727] registered taskstats version 1 [ 3.924948] Loading compiled-in X.509 certificates [ 3.926706] zswap: loaded using pool lzo/zbud [ 3.958821] Key type big_key registered [ 3.974768] Key type encrypted registered [ 3.976802] ima: No TPM chip found, activating TPM-bypass! [ 3.978658] ima: Allocated hash algorithm: sha1 [ 3.980761] ima: No architecture policies found [ 3.982839] evm: Initialising EVM extended attributes: [ 3.984965] evm: security.selinux [ 3.986418] evm: security.ima [ 3.988128] evm: security.capability [ 3.989818] evm: HMAC attrs: 0x1 [ 3.992976] rtc_cmos 00:05: setting system clock to 2026-08-17 02:38:33 UTC (1786934313) [ 4.001246] debug: unmapping init [mem 0xffffffffbb803000-0xffffffffbb9fffff] [ 4.004844] debug: unmapping init [mem 0xffffffffba582000-0xffffffffba858fff] [ 4.014181] Write protecting the kernel read-only data: 28672k [ 4.019636] debug: unmapping init [mem 0xffffffffb8c03000-0xffffffffb8dfffff] [ 4.024287] debug: unmapping init [mem 0xffffffffb9514000-0xffffffffb95fffff] [ 4.101784] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.111410] systemd[1]: Detected virtualization kvm. [ 4.114667] systemd[1]: Detected architecture x86-64. [ 4.118959] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.172967] systemd[1]: No hostname configured. [ 4.176403] systemd[1]: Set hostname to . [ 4.179036] random: systemd: uninitialized urandom read (16 bytes read) [ 4.181241] systemd[1]: Initializing machine ID from random generator. [ 4.407774] random: systemd: uninitialized urandom read (16 bytes read) [ 4.410610] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.421500] random: systemd: uninitialized urandom read (16 bytes read) [ 4.423745] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.434731] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Reached target Slices. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.420457] device-mapper: uevent: version 1.0.3 [ 5.422703] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [[ 6.375370] random: fast init done  OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.390901] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ 6.589678] scsi host0: ata_piix [ 6.605153] scsi host1: ata_piix [ 6.614707] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.617210] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 11.495157] random: crng init done [ 11.496693] random: 7 urandom warning(s) missed due to ratelimiting [ 11.588288] dracut-initqueue[587]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 12.451510] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 13.921170] printk: systemd: 21 output lines suppressed due to ratelimiting [ 14.424774] SELinux: Disabled at runtime. [ 14.509479] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 14.522336] systemd[1]: Detected virtualization kvm. [ 14.523991] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 15.337902] systemd[1]: initrd-switch-root.service: Succeeded. [ 15.353495] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 15.369469] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 15.378172] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 15.383530] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 15.403105] systemd[1]: Starting Journal Service... Starting Journal Service... [ 15.432475] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target I[ 15.619953] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS nitrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 16.107957] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 16.744408] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 16.873035] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 17.106517] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 17.135543] EDAC sbridge: Ver: 1.1.2 [ 19.733984] Key type dns_resolver registered [ 20.181930] NFS: Registering the id_resolver key type [ 20.183820] Key type id_resolver registered [ 20.186125] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg234-server login: [ 43.726388] libcfs: loading out-of-tree module taints kernel. [ 43.746063] Key type ._llcrypt registered [ 43.746948] Key type .llcrypt registered [ 43.786330] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 50.564630] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 51.164576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 51.170959] alg: No test for adler32 (adler32-zlib) [ 52.153212] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 52.453072] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 54.063155] Key type lgssc registered [ 54.676428] Lustre: Echo OBD driver; http://www.lustre.org/ [ 61.173197] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 73.656393] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 77.528942] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 77.534954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 78.617093] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 78.627833] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 78.664814] Lustre: lustre-MDT0000: new disk, initializing [ 78.685904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 78.692339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 80.034818] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 85.130778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 85.166420] Lustre: 6480:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 85.176799] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 85.180350] Lustre: Skipped 1 previous similar message [ 85.221174] Lustre: lustre-MDT0001: new disk, initializing [ 85.240733] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 85.249797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 85.252037] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 86.593598] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 88.874163] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 92.014520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 92.104394] Lustre: lustre-OST0000: new disk, initializing [ 92.105755] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 92.107624] Lustre: 8417:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 92.124498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 94.004866] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 97.258174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 97.261286] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 97.277906] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 99.383102] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 99.424558] Lustre: lustre-OST0001: new disk, initializing [ 99.426039] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 99.427955] Lustre: 9491:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 99.445325] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 101.224463] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 103.082274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 103.084514] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 103.099810] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 106.738358] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 113.348775] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 115.307897] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing check_logdir /tmp/testlogs/ [ 116.789243] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing yml_node [ 118.183503] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 119.052672] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 119.946560] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 120.510646] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Aug 16 22:40:28 EDT 2026 [ 126.037826] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 126.535846] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 139.231955] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 139.232419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 139.236863] Lustre: Skipped 3 previous similar messages [ 139.240129] Lustre: Skipped 2 previous similar messages [ 144.351797] Lustre: server umount lustre-MDT0000 complete [ 145.524588] LustreError: 8416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786934455 with bad export cookie 5812892116223352211 [ 145.524677] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 145.529197] LustreError: 8416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 149.471484] LustreError: 7824:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 149.471620] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 149.471896] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 149.475932] LustreError: 7824:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 149.478839] Lustre: Skipped 1 previous similar message [ 150.495474] LustreError: 9098:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 150.495610] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 150.499511] LustreError: 9098:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 150.501965] Lustre: Skipped 2 previous similar messages [ 151.707245] Lustre: server umount lustre-MDT0001 complete [ 167.391141] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 167.420706] Lustre: server umount lustre-OST0000 complete [ 174.735570] Lustre: server umount lustre-OST0001 complete [ 176.835596] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 179.862065] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 191.687152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 191.768650] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 191.779294] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 191.811569] Lustre: lustre-MDT0000: new disk, initializing [ 191.829917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 191.834580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 193.035286] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 196.961627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 196.999700] Lustre: 15034:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 197.111067] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 198.332628] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 201.840952] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 203.480775] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 205.548434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 205.648182] Lustre: lustre-OST0000: new disk, initializing [ 205.649470] Lustre: Skipped 1 previous similar message [ 205.650885] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 205.652417] Lustre: Skipped 2 previous similar messages [ 205.654597] Lustre: 17000:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 205.681758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 205.684058] Lustre: Skipped 1 previous similar message [ 207.031330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 207.034549] Lustre: Skipped 1 previous similar message [ 207.036648] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 207.071168] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 207.469205] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 211.045496] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 212.323052] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 212.390648] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 213.722422] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 213.799656] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 216.526347] Lustre: server umount lustre-OST0000 complete [ 222.688581] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 222.689171] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 222.689891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 222.695988] Lustre: Skipped 1 previous similar message [ 227.808513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 227.810597] Lustre: Skipped 2 previous similar messages [ 227.838518] Lustre: server umount lustre-MDT0000 complete [ 229.044096] LustreError: 15026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786934538 with bad export cookie 5812892116223353730 [ 229.045867] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 229.049183] LustreError: 15026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 238.608231] Lustre: server umount lustre-MDT0001 complete [ 239.727262] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 22:42:27 (1786934547) [ 241.864022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 242.023531] LustreError: 19261:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 242.050687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 243.327115] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 246.030671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 247.519652] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 249.900978] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 251.718479] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 253.887500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 255.939903] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 258.385668] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 271.327885] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 271.331321] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 271.336228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 272.355259] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 273.367213] Lustre: server umount lustre-OST0000 complete [ 276.694273] Lustre: server umount lustre-MDT0000 complete [ 276.959736] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 276.962518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 276.966606] Lustre: Skipped 1 previous similar message [ 276.968325] LustreError: 19261:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 277.936467] LustreError: 19243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786934587 with bad export cookie 5812892116223354612 [ 277.939424] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 277.942722] LustreError: 19243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 278.056158] Lustre: server umount lustre-MDT0001 complete [ 281.648992] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 282.687813] Key type lgssc unregistered [ 282.815480] LNet: 22812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 282.817868] LNetError: 22812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 282.826271] LNet: Removed LNI 192.168.202.134@tcp [ 283.153095] Key type .llcrypt unregistered [ 283.153866] Key type ._llcrypt unregistered [ 291.844610] Key type ._llcrypt registered [ 291.845783] Key type .llcrypt registered [ 291.882138] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 292.203789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 292.258380] alg: No test for adler32 (adler32-zlib) [ 293.132402] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 293.218334] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 294.799183] Key type lgssc registered [ 295.166676] Lustre: Echo OBD driver; http://www.lustre.org/ [ 297.702949] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 297.709747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 298.853236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 300.098836] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 302.868820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 303.001208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 304.466795] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 306.882913] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 308.640916] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 310.723188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 310.828914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 312.583816] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 314.856630] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 324.133556] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 333.280103] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 333.282096] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 333.285524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 334.305360] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 334.306378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 334.307196] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 334.312754] Lustre: Skipped 1 previous similar message [ 335.377136] Lustre: server umount lustre-OST0000 complete [ 338.911700] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 338.914017] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 338.918035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 342.679319] Lustre: server umount lustre-MDT0000 complete [ 343.823318] LustreError: 24182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786934653 with bad export cookie 18376928417443938242 [ 343.826132] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 343.827805] LustreError: 24182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 343.945035] Lustre: server umount lustre-MDT0001 complete [ 347.162196] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 348.127506] Key type lgssc unregistered [ 348.252453] LNet: 28196:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 348.255256] LNetError: 28196:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 348.265238] LNet: Removed LNI 192.168.202.134@tcp [ 348.538061] Key type .llcrypt unregistered [ 348.538933] Key type ._llcrypt unregistered [ 356.759463] Key type ._llcrypt registered [ 356.760405] Key type .llcrypt registered [ 356.794176] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 357.168117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 357.175806] alg: No test for adler32 (adler32-zlib) [ 358.020420] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 358.097944] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 359.679139] Key type lgssc registered [ 360.033424] Lustre: Echo OBD driver; http://www.lustre.org/ [ 362.481337] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 362.488317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 363.621673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 364.917694] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 367.569769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 367.686859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 369.002486] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 371.291466] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 373.040816] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 377.113396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 377.213488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 378.954272] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 381.175970] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 388.516765] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 393.697558] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 393.698341] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 393.700034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 393.700721] LustreError: Skipped 1 previous similar message [ 393.705653] Lustre: Skipped 1 previous similar message [ 394.828649] Lustre: server umount lustre-OST0000 complete [ 398.104787] Lustre: server umount lustre-MDT0000 complete [ 398.304798] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 398.308402] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 398.313641] LustreError: 29585:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 398.319215] LustreError: 29585:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 398.816919] LustreError: 29580:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 399.291818] LustreError: 29610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786934708 with bad export cookie 16855916408763271803 [ 399.294865] LustreError: 29610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 399.295059] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 399.404185] Lustre: server umount lustre-MDT0001 complete [ 402.695419] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 403.650671] Key type lgssc unregistered [ 403.775388] LNet: 33481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 403.778526] LNetError: 33481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 403.790237] LNet: Removed LNI 192.168.202.134@tcp [ 404.071176] Key type .llcrypt unregistered [ 404.072573] Key type ._llcrypt unregistered [ 412.206109] Key type ._llcrypt registered [ 412.207636] Key type .llcrypt registered [ 412.244328] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 412.568954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 412.612311] alg: No test for adler32 (adler32-zlib) [ 413.468283] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 413.546115] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 415.127134] Key type lgssc registered [ 415.498583] Lustre: Echo OBD driver; http://www.lustre.org/ [ 418.021984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 418.028815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 419.170890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 420.424326] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 423.189192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 423.309479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 424.633274] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 426.887517] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 428.634147] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 430.733846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 430.833060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 432.602407] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 434.828629] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 445.158579] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 448.479693] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 448.481783] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 448.485538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 450.529068] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 450.530548] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 450.530561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 453.963872] Lustre: server umount lustre-OST0000 complete [ 455.648109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 455.648794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 455.653090] Lustre: Skipped 1 previous similar message [ 455.655461] Lustre: Skipped 1 previous similar message [ 460.768167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 460.769889] Lustre: Skipped 1 previous similar message [ 461.274470] Lustre: server umount lustre-MDT0000 complete [ 462.375053] LustreError: 37466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786934771 with bad export cookie 9451776242444575371 [ 462.377885] LustreError: 37466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 462.378413] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 462.480639] Lustre: server umount lustre-MDT0001 complete [ 465.653914] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 466.580583] Key type lgssc unregistered [ 466.709316] LNet: 38419:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 466.711896] LNetError: 38419:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 466.724244] LNet: Removed LNI 192.168.202.134@tcp [ 466.997210] Key type .llcrypt unregistered [ 466.998134] Key type ._llcrypt unregistered [ 475.207690] Key type ._llcrypt registered [ 475.208839] Key type .llcrypt registered [ 475.245504] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 475.682556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 475.688683] alg: No test for adler32 (adler32-zlib) [ 476.544567] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 476.630720] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 478.207158] Key type lgssc registered [ 478.561297] Lustre: Echo OBD driver; http://www.lustre.org/ [ 481.272039] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 481.277961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 482.417391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 483.768738] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 486.520232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 486.636128] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 487.904671] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 490.171352] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 491.921231] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 494.035121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 494.150598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 495.999456] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 498.310556] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 507.492932] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 512.992942] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 512.993598] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 512.994590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 512.994927] LustreError: Skipped 1 previous similar message [ 512.999628] Lustre: Skipped 1 previous similar message [ 513.931772] Lustre: lustre-OST0000: Not available for connect from 192.168.202.34@tcp (stopping) [ 513.934537] Lustre: Skipped 1 previous similar message [ 518.112728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 518.115323] Lustre: Skipped 1 previous similar message [ 523.232410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 523.234719] Lustre: Skipped 2 previous similar messages [ 525.279099] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 525.324755] Lustre: server umount lustre-OST0000 complete [ 537.971532] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 537.976262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 537.977856] Lustre: Skipped 2 previous similar messages [ 544.148223] Lustre: server umount lustre-MDT0000 complete [ 545.347099] LustreError: 39790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786934854 with bad export cookie 1597464409509088071 [ 545.349803] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 545.350828] LustreError: 39790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 551.534828] Lustre: server umount lustre-MDT0001 complete [ 554.947375] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 555.866452] Key type lgssc unregistered [ 555.986291] LNet: 43537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 555.988683] LNetError: 43537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 556.000270] LNet: Removed LNI 192.168.202.134@tcp [ 556.272820] Key type .llcrypt unregistered [ 556.273714] Key type ._llcrypt unregistered [ 564.568281] Key type ._llcrypt registered [ 564.569169] Key type .llcrypt registered [ 564.604134] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 564.920598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 564.975047] alg: No test for adler32 (adler32-zlib) [ 565.827430] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 565.921763] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 567.495186] Key type lgssc registered [ 567.865023] Lustre: Echo OBD driver; http://www.lustre.org/ [ 570.409148] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 570.416922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 571.566979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 572.985622] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 575.728223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 575.865541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 577.200216] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 579.627504] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 581.492872] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 583.807615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 583.939254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 585.939535] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 588.426707] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 597.286095] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 601.568606] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 601.570684] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 601.574176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 602.594314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 602.595516] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 604.044359] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.34@tcp (stopping) [ 604.046510] Lustre: Skipped 2 previous similar messages [ 607.326382] Lustre: server umount lustre-MDT0000 complete [ 607.712745] LustreError: 44926:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 607.718611] LustreError: 44926:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 608.563204] LustreError: 46767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786934918 with bad export cookie 6066955537510553327 [ 608.567664] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 608.568094] LustreError: 46767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 609.154330] LustreError: 44923:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 609.162350] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.34@tcp (stopping) [ 612.831922] LustreError: 47415:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 612.832187] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 612.840529] Lustre: Skipped 1 previous similar message [ 613.856256] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 613.858194] Lustre: Skipped 1 previous similar message [ 614.769142] Lustre: server umount lustre-MDT0001 complete [ 632.526162] Lustre: server umount lustre-OST0000 complete [ 637.281847] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 638.177973] Key type lgssc unregistered [ 638.302359] LNet: 48807:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 638.304688] LNetError: 48807:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 638.311265] LNet: Removed LNI 192.168.202.134@tcp [ 638.613873] Key type .llcrypt unregistered [ 638.614912] Key type ._llcrypt unregistered [ 648.291491] Key type ._llcrypt registered [ 648.292609] Key type .llcrypt registered [ 648.331630] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 648.773759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 648.828645] alg: No test for adler32 (adler32-zlib) [ 649.707168] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 649.804508] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 651.383238] Key type lgssc registered [ 651.756879] Lustre: Echo OBD driver; http://www.lustre.org/ [ 654.345716] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 654.352531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 669.791297] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 674.911224] LustreError: 50173:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.134@tcp: failed processing log, type 4: rc = -110 [ 701.535268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 703.204094] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 705.335956] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 869.584212] Lustre: server umount lustre-OST0000 complete [ 874.886046] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 875.904440] Key type lgssc unregistered [ 876.030350] LNet: 51845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 876.032574] LNetError: 51845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 876.043280] LNet: Removed LNI 192.168.202.134@tcp [ 876.347464] Key type .llcrypt unregistered [ 876.348516] Key type ._llcrypt unregistered [ 884.992730] Key type ._llcrypt registered [ 884.993935] Key type .llcrypt registered [ 885.033161] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 885.366049] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 885.415334] alg: No test for adler32 (adler32-zlib) [ 886.264962] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 886.360595] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 887.943115] Key type lgssc registered [ 888.307767] Lustre: Echo OBD driver; http://www.lustre.org/ [ 890.831101] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 890.837492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 891.980058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 893.332481] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 896.080581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 896.198412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 897.517856] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 899.876716] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 901.632169] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 903.813694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 903.929235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 905.797835] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 908.091830] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 911.329416] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 911.333549] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 911.333862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 914.890318] Lustre: server umount lustre-OST0000 complete [ 916.448481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 916.454147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 916.455991] Lustre: Skipped 1 previous similar message [ 921.567970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 921.569589] Lustre: Skipped 1 previous similar message [ 922.195634] Lustre: server umount lustre-MDT0000 complete [ 923.327060] LustreError: 53215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786935232 with bad export cookie 7126483913111749741 [ 923.330188] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 923.332181] LustreError: 53215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 923.446288] Lustre: server umount lustre-MDT0001 complete [ 927.058920] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 928.029713] Key type lgssc unregistered [ 928.158378] LNet: 56766:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 928.160966] LNetError: 56766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 928.172272] LNet: Removed LNI 192.168.202.134@tcp [ 928.480613] Key type .llcrypt unregistered [ 928.481723] Key type ._llcrypt unregistered [ 937.120344] Key type ._llcrypt registered [ 937.121295] Key type .llcrypt registered [ 937.159183] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 937.513266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 937.522940] alg: No test for adler32 (adler32-zlib) [ 938.379075] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 938.462262] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 940.047159] Key type lgssc registered [ 940.404118] Lustre: Echo OBD driver; http://www.lustre.org/ [ 942.871455] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 942.877325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 958.303337] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 963.423347] LustreError: 58129:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.134@tcp: failed processing log, type 4: rc = -110 [ 990.047180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 991.879507] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 994.226253] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 996.330031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 996.485945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 996.510910] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 997.782861] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1000.537490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1000.662360] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1001.938238] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1004.235275] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1005.984279] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1007.071864] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1007.073923] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1007.077394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1010.146024] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1010.147106] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1012.750098] Lustre: server umount lustre-OST0000 complete [ 1026.789095] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1026.794347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1026.796682] Lustre: Skipped 1 previous similar message [ 1029.856771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1029.856897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1029.861631] Lustre: Skipped 1 previous similar message [ 1035.232231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1035.234975] Lustre: Skipped 1 previous similar message [ 1041.375081] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1041.429189] Lustre: server umount lustre-MDT0000 complete [ 1042.583169] LustreError: 58136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786935352 with bad export cookie 3305666266494937632 [ 1042.586130] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1042.588482] LustreError: 58136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1048.744441] Lustre: server umount lustre-MDT0001 complete [ 1052.291939] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1053.341116] Key type lgssc unregistered [ 1053.468447] LNet: 61877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1053.471363] LNetError: 61877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1053.480284] LNet: Removed LNI 192.168.202.134@tcp [ 1053.806293] Key type .llcrypt unregistered [ 1053.808052] Key type ._llcrypt unregistered [ 1062.390203] Key type ._llcrypt registered [ 1062.391989] Key type .llcrypt registered [ 1062.427928] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1062.734804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1062.780665] alg: No test for adler32 (adler32-zlib) [ 1063.626446] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 1063.707079] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 1065.287135] Key type lgssc registered [ 1065.628951] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1068.124832] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1068.131207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1069.261830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1070.476982] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1073.172292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1073.296808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1074.579875] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1076.878362] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1078.707703] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1080.798137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1080.908178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1082.748573] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1084.992346] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1095.271977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1103.839904] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1103.841806] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1103.845278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1105.889113] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1105.891393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1105.891400] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1105.898501] Lustre: Skipped 1 previous similar message [ 1107.976612] Lustre: server umount lustre-OST0000 complete [ 1109.471820] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1109.473850] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1109.477395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1115.288612] Lustre: server umount lustre-MDT0000 complete [ 1116.127973] LustreError: 63265:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1116.132439] LustreError: 63265:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1116.508603] LustreError: 65104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786935426 with bad export cookie 5003795494293629267 [ 1116.512416] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1116.618758] Lustre: server umount lustre-MDT0001 complete [ 1120.057854] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1121.058287] Key type lgssc unregistered [ 1121.178474] LNet: 66816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1121.181551] LNetError: 66816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1121.192268] LNet: Removed LNI 192.168.202.134@tcp [ 1121.476454] Key type .llcrypt unregistered [ 1121.477574] Key type ._llcrypt unregistered [ 1131.002322] Key type ._llcrypt registered [ 1131.004096] Key type .llcrypt registered [ 1131.046755] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1131.518299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1131.569096] alg: No test for adler32 (adler32-zlib) [ 1132.432574] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 1132.522655] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 1134.103155] Key type lgssc registered [ 1134.506901] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1137.447800] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 22:57:25 (1786935445) [ 1139.530961] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1139.537131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1140.680117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1141.951282] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1144.701174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1144.825678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1146.101257] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1148.367098] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1150.072512] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1152.459196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1152.586322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1154.626867] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1157.280068] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1165.922888] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1169.825996] Lustre: Failing over lustre-MDT0000 [ 1170.028720] Lustre: server umount lustre-MDT0000 complete [ 1170.399712] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1170.401710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1170.405083] LustreError: 69932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1170.408767] LustreError: 69932:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1171.424338] LustreError: 68495:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1171.425791] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1172.873302] LustreError: 68491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1172.877494] LustreError: 68491:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1172.909882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1172.953760] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1173.008827] Lustre: *** cfs_fail_loc=135, val=0*** [ 1173.055265] LustreError: 71598:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 1173.057720] LustreError: 71598:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.134@tcp: cfg command failed: rc = -2 [ 1173.060236] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1173.060236] [ 1173.062932] LustreError: MGC192.168.202.134@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1173.067043] LustreError: 71586:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1173.069421] LustreError: 71586:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 1173.072375] LustreError: 71586:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 1173.074211] LustreError: 71586:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1173.084945] Lustre: server umount lustre-MDT0000 complete [ 1173.086161] LustreError: 71586:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1175.312643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1175.360785] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1175.460446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1175.493497] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1176.741758] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1177.986460] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1180.647170] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1180.659384] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1180.681959] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1185.760850] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1185.760891] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1185.763019] LustreError: Skipped 1 previous similar message [ 1185.765908] Lustre: Skipped 1 previous similar message [ 1185.767718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1185.769689] Lustre: Skipped 1 previous similar message [ 1187.659256] Lustre: server umount lustre-OST0000 complete [ 1190.881051] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1190.881056] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1190.885700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1190.889322] Lustre: Skipped 1 previous similar message [ 1196.000291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1196.003426] Lustre: Skipped 1 previous similar message [ 1201.120241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1201.122788] Lustre: Skipped 1 previous similar message [ 1203.167109] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1203.228333] Lustre: server umount lustre-MDT0000 complete [ 1204.388198] LustreError: 69227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786935513 with bad export cookie 6089161088470996388 [ 1204.391204] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1204.392383] LustreError: 69227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1204.504394] Lustre: server umount lustre-MDT0001 complete [ 1208.014324] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1208.984929] Key type lgssc unregistered [ 1209.107385] LNet: 73376:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1209.109450] LNetError: 73376:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1209.118245] LNet: Removed LNI 192.168.202.134@tcp [ 1209.407250] Key type .llcrypt unregistered [ 1209.408275] Key type ._llcrypt unregistered [ 1217.778037] Key type ._llcrypt registered [ 1217.778828] Key type .llcrypt registered [ 1217.812541] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1218.123392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1218.164617] alg: No test for adler32 (adler32-zlib) [ 1219.013849] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 1219.095505] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 1220.671122] Key type lgssc registered [ 1221.033671] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1223.476350] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1223.482209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1224.620819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1225.873900] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1228.539231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1228.660626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1229.877188] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1232.108571] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1233.810437] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1235.989576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1236.097893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1237.857517] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1240.177416] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1250.471836] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1252.749196] Lustre: Failing over lustre-MDT0000 [ 1252.847982] Lustre: server umount lustre-MDT0000 complete [ 1254.368993] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1254.371041] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1254.374663] LustreError: 75493:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1254.378140] LustreError: 75493:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1255.798432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1255.816158] LustreError: 74761:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1255.837783] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1255.892270] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1255.904307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 1255.905973] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1255.907026] Lustre: Skipped 1 previous similar message [ 1255.910032] Lustre: Skipped 1 previous similar message [ 1255.950918] LustreError: 77871:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 1255.954953] LustreError: 77871:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.134@tcp: cfg command failed: rc = -9 [ 1255.959420] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1255.959420] [ 1255.964194] LustreError: MGC192.168.202.134@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1255.967985] LustreError: 77859:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1255.970171] LustreError: 77859:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 1255.972708] LustreError: 77859:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 1255.974341] LustreError: 77859:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1255.984272] Lustre: server umount lustre-MDT0000 complete [ 1255.985401] LustreError: 77859:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1258.053198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1258.090446] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1258.178163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1258.209309] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1259.410440] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1260.930643] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1263.590154] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1263.597373] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1263.613662] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1265.119908] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1265.122157] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1265.126482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1265.128293] Lustre: Skipped 1 previous similar message [ 1268.706696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1268.706710] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1268.712447] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1270.599735] Lustre: server umount lustre-OST0000 complete [ 1273.824935] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1273.825181] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1273.826023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1273.826028] Lustre: Skipped 1 previous similar message [ 1273.828069] Lustre: Skipped 1 previous similar message [ 1278.944261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1278.947210] Lustre: Skipped 2 previous similar messages [ 1286.111133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1286.165080] Lustre: server umount lustre-MDT0000 complete [ 1287.364329] LustreError: 74745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786935596 with bad export cookie 6671073750140044216 [ 1287.368869] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1287.368921] LustreError: 74745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1287.488708] Lustre: server umount lustre-MDT0001 complete [ 1290.978743] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1291.987677] Key type lgssc unregistered [ 1292.106314] LNet: 79649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1292.108402] LNetError: 79649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1292.117300] LNet: Removed LNI 192.168.202.134@tcp [ 1292.418956] Key type .llcrypt unregistered [ 1292.420403] Key type ._llcrypt unregistered [ 1296.348971] Key type ._llcrypt registered [ 1296.351459] Key type .llcrypt registered [ 1296.389181] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1296.511490] Key type .llcrypt unregistered [ 1296.513583] Key type ._llcrypt unregistered [ 1304.925853] Key type ._llcrypt registered [ 1304.926894] Key type .llcrypt registered [ 1304.967350] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1305.342037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1305.367321] alg: No test for adler32 (adler32-zlib) [ 1306.223529] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 1306.316748] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 1307.895106] Key type lgssc registered [ 1308.249823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1310.778131] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1310.783635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1311.919538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1313.162483] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1315.887155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1316.004618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1317.289539] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1319.587221] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1321.272803] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1323.377722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1323.482659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1325.302186] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1327.591801] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1336.807066] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1340.895892] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1340.899036] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1340.904585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1341.921367] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1341.924560] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1341.924679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1341.932145] Lustre: Skipped 1 previous similar message [ 1346.956736] Lustre: server umount lustre-OST0000 complete [ 1352.160911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1352.161037] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1352.161554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1352.172118] Lustre: Skipped 1 previous similar message [ 1357.280227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1357.282045] Lustre: Skipped 2 previous similar messages [ 1362.399092] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1362.400072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1362.402849] Lustre: Skipped 1 previous similar message [ 1362.450964] Lustre: server umount lustre-MDT0000 complete [ 1363.634142] LustreError: 81512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786935673 with bad export cookie 12420641075879229758 [ 1363.636956] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1363.637232] LustreError: 81512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1363.750395] Lustre: server umount lustre-MDT0001 complete [ 1367.196440] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1368.237791] Key type lgssc unregistered [ 1368.356352] LNet: 85100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1368.358625] LNetError: 85100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1368.371285] LNet: Removed LNI 192.168.202.134@tcp [ 1368.664859] Key type .llcrypt unregistered [ 1368.665961] Key type ._llcrypt unregistered [ 1377.109979] Key type ._llcrypt registered [ 1377.111962] Key type .llcrypt registered [ 1377.149475] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1377.476605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1377.519068] alg: No test for adler32 (adler32-zlib) [ 1378.378646] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 1378.477214] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 1380.063125] Key type lgssc registered [ 1380.417981] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1382.948088] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1382.958810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1384.102377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1385.344556] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1388.025262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1388.137798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1389.420839] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1391.760928] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1393.468672] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1395.605145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1395.714084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1397.559186] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1399.789753] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1410.086698] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 1413.087818] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1413.089898] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1413.093324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1415.138346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1415.138442] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1415.143139] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1418.633162] Lustre: server umount lustre-OST0000 complete [ 1420.256903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1420.257619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1420.259989] Lustre: Skipped 1 previous similar message [ 1420.263276] Lustre: Skipped 2 previous similar messages [ 1425.376093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1425.377681] Lustre: Skipped 1 previous similar message [ 1430.496247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1430.498013] Lustre: Skipped 1 previous similar message [ 1434.079084] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1434.136101] Lustre: server umount lustre-MDT0000 complete [ 1435.353360] LustreError: 88330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786935744 with bad export cookie 13140606503742831365 [ 1435.359015] LustreError: 88330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1435.359212] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1435.482235] Lustre: server umount lustre-MDT0001 complete [ 1438.794657] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1439.719950] Key type lgssc unregistered [ 1439.843375] LNet: 90042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1439.845570] LNetError: 90042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1439.855279] LNet: Removed LNI 192.168.202.134@tcp [ 1440.129559] Key type .llcrypt unregistered [ 1440.130553] Key type ._llcrypt unregistered [ 1448.294061] Key type ._llcrypt registered [ 1448.295118] Key type .llcrypt registered [ 1448.333512] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1448.762137] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1448.778880] alg: No test for adler32 (adler32-zlib) [ 1449.618395] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 1449.701451] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 1451.279112] Key type lgssc registered [ 1451.619956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1454.100459] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1454.107217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1455.241719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1456.537458] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1459.279116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1459.396302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1460.726793] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1463.087559] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1464.839234] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1466.921197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1467.026339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1468.789723] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1471.018106] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1480.358148] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 1484.768130] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1484.771777] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1484.776293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1485.792923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1485.793576] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1485.794266] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1490.701370] Lustre: server umount lustre-OST0000 complete [ 1495.519811] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1495.521825] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1495.525044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1495.526731] Lustre: Skipped 1 previous similar message [ 1501.152123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1501.154628] Lustre: Skipped 3 previous similar messages [ 1506.271168] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1506.272664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1506.276332] Lustre: Skipped 1 previous similar message [ 1506.340945] Lustre: server umount lustre-MDT0000 complete [ 1507.562352] LustreError: 91411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786935817 with bad export cookie 15304112728149108607 [ 1507.565170] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1507.566099] LustreError: 91411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1507.700539] Lustre: server umount lustre-MDT0001 complete [ 1511.327447] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1512.342643] Key type lgssc unregistered [ 1512.482440] LNet: 94986:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1512.485755] LNetError: 94986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1512.495305] LNet: Removed LNI 192.168.202.134@tcp [ 1512.899218] Key type .llcrypt unregistered [ 1512.900281] Key type ._llcrypt unregistered [ 1521.270102] Key type ._llcrypt registered [ 1521.271993] Key type .llcrypt registered [ 1521.309713] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1521.623899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1521.659132] alg: No test for adler32 (adler32-zlib) [ 1522.513648] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 1522.608476] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 1524.191125] Key type lgssc registered [ 1524.527140] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1526.965266] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1526.972047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1542.431336] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1547.551293] LustreError: 96348:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.134@tcp: failed processing log, type 4: rc = -110 [ 1574.175169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1576.035309] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1578.407066] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1580.246248] Lustre: server umount lustre-OST0000 complete [ 1582.851174] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 23:04:50 (1786935890) [ 1584.968514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1585.117205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1586.438229] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1589.188794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1589.311637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1590.677894] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1593.019555] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1594.837136] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1597.070039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1597.186365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1599.117796] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1601.503930] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1609.511387] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 1619.938057] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1619.938456] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1619.940261] LustreError: Skipped 1 previous similar message [ 1619.946173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1619.948399] Lustre: Skipped 1 previous similar message [ 1621.459535] Lustre: server umount lustre-OST0000 complete [ 1625.056823] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1625.057495] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1625.062183] Lustre: Skipped 1 previous similar message [ 1625.063917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1625.065636] Lustre: Skipped 1 previous similar message [ 1628.757595] Lustre: server umount lustre-MDT0000 complete [ 1629.981785] LustreError: 97964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786935939 with bad export cookie 5901990056793416401 [ 1629.985545] LustreError: 97964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1629.986651] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1630.112761] Lustre: server umount lustre-MDT0001 complete [ 1633.497182] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1634.491651] Key type lgssc unregistered [ 1634.618286] LNet: 101694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1634.620680] LNetError: 101694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1634.626297] LNet: Removed LNI 192.168.202.134@tcp [ 1634.907298] Key type .llcrypt unregistered [ 1634.908349] Key type ._llcrypt unregistered [ 1648.683487] Key type ._llcrypt registered [ 1648.684529] Key type .llcrypt registered [ 1648.719211] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1649.069065] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1649.133040] alg: No test for adler32 (adler32-zlib) [ 1649.986556] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 1650.073313] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 1651.655121] Key type lgssc registered [ 1651.966597] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1654.379882] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1654.385996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1655.521652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1656.834785] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1659.517428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1659.634364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1660.955488] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1663.263525] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1665.097180] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1667.246505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1667.355046] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1669.135170] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1671.508935] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1680.678027] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 1685.986290] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1685.986840] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1685.987123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1685.989895] Lustre: Skipped 1 previous similar message [ 1690.512650] Lustre: server umount lustre-OST0000 complete [ 1695.711943] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1695.714257] LustreError: Skipped 1 previous similar message [ 1695.715563] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1695.720216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1695.721980] Lustre: Skipped 1 previous similar message [ 1697.883361] Lustre: server umount lustre-MDT0000 complete [ 1699.026270] LustreError: 103836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786936008 with bad export cookie 12113768224796872393 [ 1699.029243] LustreError: 103836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1699.029289] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1699.144832] Lustre: server umount lustre-MDT0001 complete [ 1702.553799] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1703.592366] Key type lgssc unregistered [ 1703.725375] LNet: 107410:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1703.727706] LNetError: 107410:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1703.741341] LNet: Removed LNI 192.168.202.134@tcp [ 1704.017566] Key type .llcrypt unregistered [ 1704.019480] Key type ._llcrypt unregistered [ 1710.948743] Key type ._llcrypt registered [ 1710.949621] Key type .llcrypt registered [ 1710.987601] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1711.367585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1711.384686] alg: No test for adler32 (adler32-zlib) [ 1712.241395] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 1712.334221] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 1713.919173] Key type lgssc registered [ 1714.270912] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1716.834714] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1716.840709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1732.255331] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1737.375295] LustreError: 108629:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.134@tcp: failed processing log, type 4: rc = -110 [ 1763.999166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1765.739222] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1767.978332] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1770.025544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1770.116071] LustreError: MGC192.168.202.134@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1770.119829] LustreError: 109651:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1770.121831] LustreError: 109651:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 1770.124219] LustreError: 109651:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 1770.149486] Lustre: server umount lustre-MDT0000 complete [ 1770.151352] LustreError: 109651:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1781.758680] Lustre: server umount lustre-OST0000 complete [ 1784.777393] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 1787.573590] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1798.835119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1798.954979] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1798.966180] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1799.010404] Lustre: lustre-MDT0000: new disk, initializing [ 1799.034415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1799.040685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1800.454512] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1804.346758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1804.387231] Lustre: 112436:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1804.407554] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1804.409235] Lustre: Skipped 1 previous similar message [ 1804.444916] Lustre: lustre-MDT0001: new disk, initializing [ 1804.477371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1804.490799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1804.497191] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1805.744704] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1809.372619] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1811.184919] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1813.423135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1813.521243] Lustre: lustre-OST0000: new disk, initializing [ 1813.522800] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1813.525104] Lustre: 114402:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1813.551915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1815.362790] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1816.061675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1816.065858] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1816.093370] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1819.009708] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1820.410627] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1820.482661] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1821.889338] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1821.963501] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1826.273892] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1826.274562] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1826.276642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1826.281593] Lustre: Skipped 1 previous similar message [ 1826.285014] Lustre: Skipped 1 previous similar message [ 1828.757185] Lustre: server umount lustre-OST0000 complete [ 1830.368056] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1830.370213] LustreError: Skipped 1 previous similar message [ 1830.371413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1830.375509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1831.392470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1831.392939] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1836.122473] Lustre: server umount lustre-MDT0000 complete [ 1837.304226] LustreError: 113232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786936146 with bad export cookie 690687522126467136 [ 1837.307730] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1837.309446] LustreError: 113232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1837.431340] Lustre: server umount lustre-MDT0001 complete [ 1839.990053] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 23:09:08 (1786936148) [ 1841.003876] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 1845.916162] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 1848.758397] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1859.677537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1859.755238] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1859.763663] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1859.798557] Lustre: lustre-MDT0000: new disk, initializing [ 1859.816990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1859.822404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1861.111281] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1865.038246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1865.082561] Lustre: 119293:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1865.103137] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1865.104802] Lustre: Skipped 1 previous similar message [ 1865.138417] Lustre: lustre-MDT0001: new disk, initializing [ 1865.176099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1865.178917] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1866.435961] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1870.278440] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1872.044080] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1874.243732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1874.344117] Lustre: lustre-OST0000: new disk, initializing [ 1874.345607] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1874.347717] Lustre: 121256:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1874.373633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1874.375586] Lustre: Skipped 1 previous similar message [ 1876.203220] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1878.263462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1878.266449] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1878.300276] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1879.750652] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1881.093543] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1881.161421] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1882.390106] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1882.457728] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1883.617531] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1883.618191] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1883.619496] LustreError: Skipped 1 previous similar message [ 1883.623190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1883.625632] Lustre: Skipped 2 previous similar messages [ 1885.140442] Lustre: server umount lustre-OST0000 complete [ 1888.736704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1888.737400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1888.740069] Lustre: Skipped 2 previous similar messages [ 1892.449760] Lustre: server umount lustre-MDT0000 complete [ 1893.786375] LustreError: 120085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786936203 with bad export cookie 690687522126468004 [ 1893.789151] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1893.789413] LustreError: 120085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1893.909706] Lustre: server umount lustre-MDT0001 complete [ 1896.778985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1896.924091] LustreError: 123467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1896.929854] LustreError: 123467:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1896.947183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1898.287987] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1901.129199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1902.706573] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1905.473392] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1907.461278] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1910.022524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1912.817873] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1915.980355] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1929.336117] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 1931.744147] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1931.747115] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1931.750483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1931.752045] Lustre: Skipped 1 previous similar message [ 1936.548175] Lustre: server umount lustre-OST0000 complete [ 1944.303215] Lustre: server umount lustre-MDT0000 complete [ 1945.921145] LustreError: 123449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786936255 with bad export cookie 690687522126468872 [ 1945.924067] LustreError: 123449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1945.924687] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1946.036308] Lustre: server umount lustre-MDT0001 complete [ 1949.970807] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1951.071742] Key type lgssc unregistered [ 1951.202406] LNet: 127119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1951.205272] LNetError: 127119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1951.212377] LNet: Removed LNI 192.168.202.134@tcp [ 1951.529692] Key type .llcrypt unregistered [ 1951.531060] Key type ._llcrypt unregistered [ 1958.105588] Key type ._llcrypt registered [ 1958.107555] Key type .llcrypt registered [ 1958.161232] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 1964.084105] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1964.467079] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1964.498772] alg: No test for adler32 (adler32-zlib) [ 1965.354300] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 1965.442818] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 1967.015122] Key type lgssc registered [ 1967.379244] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1981.633565] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1981.644417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1982.746493] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1982.756935] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1982.798750] Lustre: lustre-MDT0000: new disk, initializing [ 1982.819472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1982.825264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1984.379861] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1989.483150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1989.544931] Lustre: 130244:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1989.580985] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1989.584108] Lustre: Skipped 1 previous similar message [ 1989.627780] Lustre: lustre-MDT0001: new disk, initializing [ 1989.668839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1989.684724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1989.693172] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1990.883221] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1995.362487] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1997.383293] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1999.808512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1999.951740] Lustre: lustre-OST0000: new disk, initializing [ 1999.953676] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1999.956294] Lustre: 132210:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1999.986002] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2002.316618] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2002.560515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2002.564309] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2002.603892] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2006.449701] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2008.252887] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2008.338281] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2010.152179] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2010.253936] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2013.158674] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2013.159109] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2013.160860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2013.163697] LustreError: Skipped 1 previous similar message [ 2013.173954] Lustre: Skipped 1 previous similar message [ 2017.265796] Lustre: server umount lustre-OST0000 complete [ 2020.320542] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2020.327077] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2020.338536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2020.342510] Lustre: Skipped 1 previous similar message [ 2023.392499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2023.393506] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2024.759474] Lustre: server umount lustre-MDT0000 complete [ 2026.060280] LustreError: 130237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786936335 with bad export cookie 12901557212482424474 [ 2026.064465] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2026.065484] LustreError: 130237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2026.187135] Lustre: server umount lustre-MDT0001 complete [ 2029.531790] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 23:12:17 (1786936337) [ 2032.379918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2032.534690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2033.878168] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2037.518609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2037.624462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2038.918628] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2042.338493] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2044.175802] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2047.969430] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2047.969665] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2047.970638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2047.970645] Lustre: Skipped 1 previous similar message [ 2047.975951] Lustre: Skipped 1 previous similar message [ 2051.194598] Lustre: server umount lustre-MDT0000 complete [ 2052.704385] LustreError: 135359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786936362 with bad export cookie 12901557212482425342 [ 2052.708541] LustreError: 135359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2052.709309] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2052.829782] Lustre: server umount lustre-MDT0001 complete [ 2055.819983] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 23:12:43 (1786936363) [ 2058.126727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2073.567622] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2078.687728] LustreError: 137034:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.134@tcp: failed processing log, type 4: rc = -110 [ 2104.287751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2106.532343] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2109.750495] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2110.649862] Lustre: server umount lustre-OST0000 complete [ 2113.999100] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 23:13:41 (1786936421) [ 2116.923458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2117.105139] LustreError: 138478:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2117.113569] LustreError: 138478:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2118.648401] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2121.875597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2122.002201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2122.005532] Lustre: Skipped 1 previous similar message [ 2123.210918] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2126.205458] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2128.563221] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2130.909922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2133.243631] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2135.606201] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2147.809549] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2147.810297] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2147.811567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2147.811573] Lustre: Skipped 3 previous similar messages [ 2147.822563] Lustre: Skipped 2 previous similar messages [ 2153.881116] Lustre: server umount lustre-MDT0000 complete [ 2155.080238] LustreError: 140976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786936464 with bad export cookie 12901557212482425993 [ 2155.083463] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2155.084990] LustreError: 140976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2155.204250] Lustre: server umount lustre-MDT0001 complete [ 2165.592644] Lustre: server umount lustre-OST0000 complete [ 2168.252534] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 23:14:36 (1786936476) [ 2170.369175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2170.518821] LustreError: 142147:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2170.543464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2170.546520] Lustre: Skipped 1 previous similar message [ 2171.837513] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2174.756258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2176.270863] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2178.631811] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2180.475326] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2182.682507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2184.659177] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2186.963351] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2188.305286] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2196.134364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 2196.529412] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 2197.880923] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2197.960994] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2200.544693] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2200.548076] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2200.553995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2200.555469] Lustre: Skipped 5 previous similar messages [ 2204.685126] Lustre: server umount lustre-OST0000 complete [ 2212.004675] Lustre: server umount lustre-MDT0000 complete [ 2213.283155] LustreError: 142128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786936522 with bad export cookie 12901557212482427050 [ 2213.285517] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2213.287024] LustreError: 142128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2213.402440] Lustre: server umount lustre-MDT0001 complete [ 2216.324197] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 23:15:24 (1786936524) [ 2218.713149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2234.143622] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2239.263524] LustreError: 146356:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.134@tcp: failed processing log, type 4: rc = -110 [ 2264.863460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2264.868304] Lustre: Skipped 2 previous similar messages [ 2266.844420] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2269.744480] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2272.762825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2272.906936] LustreError: 146381:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2272.957633] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 2274.522371] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2277.772636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2279.115827] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2281.323222] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2283.348095] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2284.696596] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2284.772250] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2286.430383] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2286.499276] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2288.097550] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2288.102043] LustreError: Skipped 1 previous similar message [ 2288.104571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2288.111186] Lustre: Skipped 3 previous similar messages [ 2288.114362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2288.118927] Lustre: Skipped 6 previous similar messages [ 2293.556697] Lustre: server umount lustre-MDT0000 complete [ 2294.940207] LustreError: 147420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786936604 with bad export cookie 12901557212482428058 [ 2294.944426] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2294.945317] LustreError: 147420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2295.060183] Lustre: server umount lustre-MDT0001 complete [ 2306.518361] Lustre: server umount lustre-OST0000 complete [ 2308.902885] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 23:16:57 (1786936617) [ 2311.066228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2326.495843] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2331.615584] LustreError: 150548:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.134@tcp: failed processing log, type 4: rc = -110 [ 2359.425173] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2362.995759] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2366.089077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2366.288120] LustreError: 150573:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2366.369188] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 2367.838331] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2371.093538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2372.683819] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2375.279687] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2377.252478] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2379.904292] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2379.979759] Lustre: lustre-OST0001: new disk, initializing [ 2379.981703] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2379.984381] Lustre: 153419:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2382.054766] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2385.930949] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2387.330229] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2388.984135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2388.986727] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2389.018501] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2389.446311] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2390.780497] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2390.852312] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2392.032627] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2392.032861] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2392.034516] LustreError: Skipped 1 previous similar message [ 2392.041275] Lustre: Skipped 2 previous similar messages [ 2392.042881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2392.046227] Lustre: Skipped 4 previous similar messages [ 2397.574187] Lustre: server umount lustre-OST0000 complete [ 2399.202597] LustreError: 150563:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2399.206692] LustreError: 150563:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2405.078733] Lustre: server umount lustre-OST0001 complete [ 2412.588444] Lustre: server umount lustre-MDT0000 complete [ 2413.924283] LustreError: 153418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786936723 with bad export cookie 12901557212482429010 [ 2413.927700] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2414.042352] Lustre: server umount lustre-MDT0001 complete [ 2421.905454] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 23:18:49 (1786936729) [ 2422.397076] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 2422.900649] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 2423.415672] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 23:18:51 (1786936731) [ 2425.461831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2425.556672] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2425.565574] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2425.649536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2425.651411] Lustre: Skipped 6 previous similar messages [ 2427.000405] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2429.778607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2429.808163] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2429.818216] Lustre: 156985:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2431.187672] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2433.624239] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2435.391845] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2437.693433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2437.761131] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2439.556806] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2439.693501] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 2441.743477] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2442.984450] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2443.044734] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2444.225614] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2444.290183] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2449.888791] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2449.891040] LustreError: Skipped 3 previous similar messages [ 2459.103170] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2459.185518] Lustre: server umount lustre-OST0000 complete [ 2473.097465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2475.175644] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2477.324641] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2478.680953] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2480.356237] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2480.358980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2480.359129] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2480.363300] Lustre: Skipped 1 previous similar message [ 2480.776125] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2481.958398] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2482.020520] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2483.592369] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9abbc7cb8000.ost_server_uuid 50 [ 2484.045592] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9abbc7cb8000.ost_server_uuid in FULL state after 0 sec [ 2484.097322] LustreError: 156995:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.202.34@tcp) returned error from blocking AST (req@00000000b00076dd x1873738490638848 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000004120874b/0xb30b89edeb64e99e lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.34@tcp remote: 0x2120814df324630 expref: 5 pid: 156994 timeout: 2584 lvb_type: 0 lru_score: 0 lru_type: 0 [ 2484.105576] LustreError: lustre-MDT0000: A client on nid 192.168.202.34@tcp was evicted due to a lock blocking callback time out: rc -107 [ 2484.107928] LustreError: 156983:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.34@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000004120874b/0xb30b89edeb64e99e lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.34@tcp remote: 0x2120814df324630 expref: 6 pid: 156994 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 2485.730296] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2485.731276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2485.733351] Lustre: Skipped 7 previous similar messages [ 2485.734624] Lustre: Skipped 13 previous similar messages [ 2491.143259] Lustre: server umount lustre-OST0000 complete [ 2498.453953] Lustre: server umount lustre-MDT0000 complete [ 2499.598136] LustreError: 160257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786936809 with bad export cookie 12901557212482430193 [ 2499.600333] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2499.603074] LustreError: 160257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 2505.768603] Lustre: server umount lustre-MDT0001 complete [ 2508.939324] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 2509.866094] Key type lgssc unregistered [ 2509.982385] LNet: 163107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2509.984531] LNetError: 163107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2509.995294] LNet: Removed LNI 192.168.202.134@tcp [ 2510.268106] Key type .llcrypt unregistered [ 2510.269166] Key type ._llcrypt unregistered [ 2518.294223] Key type ._llcrypt registered [ 2518.295671] Key type .llcrypt registered [ 2518.340739] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 2518.750466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2518.756401] alg: No test for adler32 (adler32-zlib) [ 2519.599928] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 2519.677883] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 2521.255119] Key type lgssc registered [ 2521.573769] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2523.928575] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2523.935812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2525.068370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2526.256278] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2528.932878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2529.058311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2530.334468] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2532.581429] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2534.205271] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2536.216287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2536.319493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2538.022793] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2540.118163] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2549.672804] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 2552.780517] Lustre: Failing over lustre-MDT0000 [ 2553.045936] Lustre: server umount lustre-MDT0000 complete [ 2554.851178] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2554.852571] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2554.852577] LustreError: 166981:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2554.852590] LustreError: 166981:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2554.854940] Lustre: Skipped 2 previous similar messages [ 2559.968506] LustreError: 166980:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2559.972543] LustreError: 166980:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2563.912659] LustreError: 164476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.34@tcp arrived at 1786936873 with bad export cookie 14516344287806014504 [ 2563.916145] LustreError: 164476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2565.088436] LustreError: 164514:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2565.092462] LustreError: 164514:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2566.149565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2566.187954] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2566.298044] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2566.330758] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2567.579727] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2568.274610] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2568.276593] Lustre: lustre-MDT0000: Denying connection for new client 88f945c3-8d8c-4565-ad64-5fa34fab2072 (at 192.168.202.34@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2571.753960] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2575.328619] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2575.330736] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2575.334548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2576.866080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2576.869264] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2581.070793] Lustre: server umount lustre-OST0000 complete [ 2582.301649] LustreError: 168240:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2582.303988] Lustre: 167590:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2582.306252] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2582.312176] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2582.315831] LustreError: 167590:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000c1a25b8d x1873739072440320/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2582.320708] LustreError: 167590:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 2582.323378] LustreError: 167590:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 2582.325918] Lustre: lustre-MDT0000: Recovery over after 0:14, of 2 clients 0 recovered and 2 were evicted. [ 2582.437837] Lustre: server umount lustre-MDT0000 complete [ 2583.644588] LustreError: 166334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786936893 with bad export cookie 14516344287806014833 [ 2583.650648] LustreError: 163807:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 2583.650807] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2589.829297] Lustre: server umount lustre-MDT0001 complete [ 2593.174787] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 2594.179467] Key type lgssc unregistered [ 2594.297343] LNet: 168996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2594.299865] LNetError: 168996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2594.308284] LNet: Removed LNI 192.168.202.134@tcp [ 2594.604407] Key type .llcrypt unregistered [ 2594.605500] Key type ._llcrypt unregistered [ 2602.997353] Key type ._llcrypt registered [ 2602.998345] Key type .llcrypt registered [ 2603.039291] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 2603.544911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2603.553948] alg: No test for adler32 (adler32-zlib) [ 2604.398761] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 2604.477990] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 2606.055135] Key type lgssc registered [ 2606.445477] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2609.490956] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2609.498605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2610.672099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2612.066748] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2614.931731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2615.053255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2616.283373] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2618.525144] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2620.371184] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2622.554994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2622.662585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2624.421965] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2626.624744] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2636.002814] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 2640.351769] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2640.355049] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2640.359490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2641.377519] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2641.378566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2641.378750] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2641.387849] Lustre: Skipped 1 previous similar message [ 2644.430042] Lustre: server umount lustre-OST0000 complete [ 2645.983849] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2645.987115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2645.992647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2651.616061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2651.618214] Lustre: Skipped 3 previous similar messages [ 2651.729506] Lustre: server umount lustre-MDT0000 complete [ 2652.850096] LustreError: 170365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786936962 with bad export cookie 1745279160618539848 [ 2652.853200] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2652.854442] LustreError: 170365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2652.960309] Lustre: server umount lustre-MDT0001 complete [ 2656.528163] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 2657.479983] Key type lgssc unregistered [ 2657.596343] LNet: 173938:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2657.598471] LNetError: 173938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2657.611246] LNet: Removed LNI 192.168.202.134@tcp [ 2657.879786] Key type .llcrypt unregistered [ 2657.880814] Key type ._llcrypt unregistered [ 2669.453209] Key type ._llcrypt registered [ 2669.454215] Key type .llcrypt registered [ 2669.486622] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 2669.806413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2669.823681] alg: No test for adler32 (adler32-zlib) [ 2670.666760] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 2670.750596] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 2672.327172] Key type lgssc registered [ 2672.679792] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2675.085366] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2675.091553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2676.240829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2677.498620] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2680.127080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2680.243569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2681.480375] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2683.680148] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2685.331417] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2687.632751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2687.746271] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2689.558883] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2691.884660] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2701.095059] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 2705.375775] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2705.377781] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2705.381358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2706.400986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2706.402225] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2706.402962] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2709.516210] Lustre: server umount lustre-OST0000 complete [ 2711.007781] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2711.011030] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2711.016679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2711.018415] Lustre: Skipped 1 previous similar message [ 2716.640062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2716.641651] Lustre: Skipped 2 previous similar messages [ 2716.816820] Lustre: server umount lustre-MDT0000 complete [ 2717.967499] LustreError: 177457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786937027 with bad export cookie 16202107783058936451 [ 2717.972193] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2717.973562] LustreError: 177457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2718.112189] Lustre: server umount lustre-MDT0001 complete [ 2721.344001] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 2722.246621] Key type lgssc unregistered [ 2722.360142] LNet: 179166:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2722.362223] LNetError: 179166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2722.371284] LNet: Removed LNI 192.168.202.134@tcp [ 2722.671255] Key type .llcrypt unregistered [ 2722.673047] Key type ._llcrypt unregistered [ 2730.436076] Key type ._llcrypt registered [ 2730.437026] Key type .llcrypt registered [ 2730.468216] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 2730.824957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2730.853948] alg: No test for adler32 (adler32-zlib) [ 2731.704497] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 2731.790545] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 2733.367117] Key type lgssc registered [ 2733.690406] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2736.954253] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2736.959862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2738.084291] Lustre: *** cfs_fail_loc=135, val=0*** [ 2738.124688] LustreError: 180665:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2738.126792] LustreError: 180665:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.134@tcp: cfg command failed: rc = -2 [ 2738.129392] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2738.129392] [ 2738.132170] LustreError: MGC192.168.202.134@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2738.139213] LustreError: 180627:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2738.143291] LustreError: 180627:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2738.147149] LustreError: 180627:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2738.150357] LustreError: 180627:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2738.184414] Lustre: server umount lustre-MDT0000 complete [ 2738.185719] LustreError: 180627:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2740.548096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2740.742980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2742.149513] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2744.921542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2745.041700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2746.258330] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2748.494649] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2750.206924] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2754.913060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2754.913710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2754.916339] Lustre: Skipped 1 previous similar message [ 2754.919303] Lustre: Skipped 1 previous similar message [ 2760.160116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2760.161754] Lustre: Skipped 1 previous similar message [ 2765.279087] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2765.280125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2765.283149] Lustre: Skipped 1 previous similar message [ 2765.341646] Lustre: server umount lustre-MDT0000 complete [ 2766.483645] LustreError: 182606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786937075 with bad export cookie 15792991147849737249 [ 2766.484646] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2766.487026] LustreError: 182606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2766.597186] Lustre: server umount lustre-MDT0001 complete [ 2770.040579] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 2770.980501] Key type lgssc unregistered [ 2771.109429] LNet: 183362:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2771.111822] LNetError: 183362:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2771.122262] LNet: Removed LNI 192.168.202.134@tcp [ 2771.401852] Key type .llcrypt unregistered [ 2771.403212] Key type ._llcrypt unregistered [ 2777.737898] Key type ._llcrypt registered [ 2777.739345] Key type .llcrypt registered [ 2777.771207] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 2777.865503] Key type .llcrypt unregistered [ 2777.866517] Key type ._llcrypt unregistered [ 2785.424040] Key type ._llcrypt registered [ 2785.425112] Key type .llcrypt registered [ 2785.482851] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 2786.046795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2786.066337] alg: No test for adler32 (adler32-zlib) [ 2786.915681] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 2787.018204] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 2788.591143] Key type lgssc registered [ 2788.909548] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2791.253204] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2791.258967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2806.687675] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2811.807413] LustreError: 185472:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.134@tcp: failed processing log, type 4: rc = -110 [ 2838.431298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2840.439554] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2842.649207] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2844.852631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2845.018860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2845.039245] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2846.295210] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2849.217679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2849.343042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2850.654721] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2853.076763] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2855.041618] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2856.128818] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 2864.097486] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2864.101582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2864.104039] Lustre: Skipped 1 previous similar message [ 2869.220352] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2869.222551] Lustre: Skipped 1 previous similar message [ 2874.339895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2874.344265] Lustre: Skipped 1 previous similar message [ 2875.359087] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2875.411874] Lustre: server umount lustre-OST0000 complete [ 2879.456974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2879.457548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2879.461930] Lustre: Skipped 2 previous similar messages [ 2879.468158] Lustre: Skipped 1 previous similar message [ 2884.576870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2884.578390] Lustre: Skipped 1 previous similar message [ 2891.231117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2891.284903] Lustre: server umount lustre-MDT0000 complete [ 2892.461956] LustreError: 186534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786937201 with bad export cookie 6082693881378026748 [ 2892.464214] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2892.464788] LustreError: 186534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2892.587850] Lustre: server umount lustre-MDT0001 complete [ 2895.721293] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 2896.608961] Key type lgssc unregistered [ 2896.729316] LNet: 189393:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2896.731242] LNetError: 189393:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2896.739263] LNet: Removed LNI 192.168.202.134@tcp [ 2897.010171] Key type .llcrypt unregistered [ 2897.011224] Key type ._llcrypt unregistered [ 2906.288086] Key type ._llcrypt registered [ 2906.289022] Key type .llcrypt registered [ 2906.321892] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 2906.649209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2906.684167] alg: No test for adler32 (adler32-zlib) [ 2907.520064] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 2907.595761] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 2909.175115] Key type lgssc registered [ 2909.506063] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2911.915750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2911.922380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2913.073052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2914.294667] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2916.970492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2917.082531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2918.361936] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2920.667251] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2922.453943] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2924.540636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2924.651991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2926.521204] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2928.833845] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2937.955272] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 2940.442880] Lustre: Failing over lustre-MDT0000 [ 2940.527704] Lustre: server umount lustre-MDT0000 complete [ 2942.944507] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2942.949272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2942.962799] LustreError: 190781:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2942.971761] LustreError: 190781:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2943.459123] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2943.469981] Lustre: Skipped 1 previous similar message [ 2943.474894] LustreError: 190778:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2943.483972] LustreError: 190778:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2948.576607] LustreError: 190781:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2948.583477] LustreError: 190781:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2953.698875] LustreError: 190782:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2953.706920] LustreError: 190782:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2954.026682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2954.087824] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2954.211622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2954.255309] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2954.628153] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2955.695089] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2957.870786] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 2959.335089] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2959.344728] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2959.368483] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 2961.348743] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 2967.009960] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2967.013691] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2967.019211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2968.033289] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2968.039611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2969.574606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2969.581166] Lustre: Skipped 1 previous similar message [ 2970.710037] Lustre: server umount lustre-OST0000 complete [ 2974.688715] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2974.688808] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2974.689785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2974.692355] Lustre: Skipped 2 previous similar messages [ 2978.212180] Lustre: server umount lustre-MDT0000 complete [ 2979.686905] LustreError: 190764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786937289 with bad export cookie 4048988506411871500 [ 2979.692877] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2979.694190] LustreError: 190764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2979.839498] Lustre: server umount lustre-MDT0001 complete [ 2984.575424] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 2986.103848] Key type lgssc unregistered [ 2986.264522] LNet: 195630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2986.269474] LNetError: 195630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2986.279476] LNet: Removed LNI 192.168.202.134@tcp [ 2986.719793] Key type .llcrypt unregistered [ 2986.721040] Key type ._llcrypt unregistered [ 2996.611209] Key type ._llcrypt registered [ 2996.612185] Key type .llcrypt registered [ 2996.644927] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 2996.972771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2997.031257] alg: No test for adler32 (adler32-zlib) [ 2997.876254] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 2997.956789] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 2999.535157] Key type lgssc registered [ 2999.960104] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3002.957377] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3002.965447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3004.121180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3005.518288] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3008.352749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3008.466219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3009.776541] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3012.030663] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3013.722600] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3015.860608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3015.968672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3017.770124] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3020.009354] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3029.292071] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 3031.028458] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3035.537287] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 3046.098661] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3055.074819] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3055.077809] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3055.079193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3055.083505] Lustre: Skipped 1 previous similar message [ 3060.192602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3060.195370] Lustre: Skipped 2 previous similar messages [ 3060.298254] Lustre: server umount lustre-OST0000 complete [ 3064.799803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3064.801918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3064.805623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3067.795983] Lustre: server umount lustre-MDT0000 complete [ 3068.941233] LustreError: 196998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786937378 with bad export cookie 5880074372846375321 [ 3068.944964] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3068.946339] LustreError: 196998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3069.056933] Lustre: server umount lustre-MDT0001 complete [ 3072.304603] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 3073.326308] Key type lgssc unregistered [ 3073.452277] LNet: 200882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3073.455235] LNetError: 200882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3073.465303] LNet: Removed LNI 192.168.202.134@tcp [ 3073.768303] Key type .llcrypt unregistered [ 3073.769440] Key type ._llcrypt unregistered [ 3080.825150] Key type ._llcrypt registered [ 3080.826560] Key type .llcrypt registered [ 3080.864039] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 3081.295254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3081.301761] alg: No test for adler32 (adler32-zlib) [ 3082.146111] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 3082.231815] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 3083.807113] Key type lgssc registered [ 3084.117761] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3086.501570] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3086.509163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3087.656983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3089.059972] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3091.930730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3092.062608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3093.340652] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3095.698116] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3097.526371] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3099.921853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3100.044213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3101.939724] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3104.999964] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3114.406596] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 3117.193736] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 3128.526376] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3134.841889] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 3139.408899] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 3150.308833] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3150.310451] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3150.311805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3150.327330] Lustre: Skipped 1 previous similar message [ 3155.426125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3155.429974] Lustre: Skipped 2 previous similar messages [ 3155.552930] Lustre: server umount lustre-OST0000 complete [ 3158.496480] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3158.498429] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3158.501857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3160.544061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3160.544746] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3163.499581] Lustre: server umount lustre-MDT0000 complete [ 3164.638286] LustreError: 204821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786937474 with bad export cookie 14776939438504091057 [ 3164.641208] LustreError: 204821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3164.641759] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3164.747907] Lustre: server umount lustre-MDT0001 complete [ 3168.285602] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 3169.264415] Key type lgssc unregistered [ 3169.388388] LNet: 207457:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3169.392783] LNetError: 207457:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3169.401274] LNet: Removed LNI 192.168.202.134@tcp [ 3169.753789] Key type .llcrypt unregistered [ 3169.754748] Key type ._llcrypt unregistered [ 3179.237841] Key type ._llcrypt registered [ 3179.238876] Key type .llcrypt registered [ 3179.277912] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 3179.634289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3179.664567] alg: No test for adler32 (adler32-zlib) [ 3180.515942] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 3180.597115] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 3182.191118] Key type lgssc registered [ 3182.549752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3185.301593] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3185.312469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3186.481351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3187.931388] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3191.045956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3191.210324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3192.985431] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3195.933568] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3198.072165] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3200.501231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3200.613538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3202.318035] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3204.450125] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3210.853141] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 3221.474327] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3221.474712] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3221.475371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3221.482034] Lustre: Skipped 1 previous similar message [ 3223.820926] Lustre: server umount lustre-OST0000 complete [ 3226.592488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3226.596314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3226.598093] Lustre: Skipped 2 previous similar messages [ 3231.712852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3231.715959] Lustre: Skipped 1 previous similar message [ 3236.833296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3236.838334] Lustre: Skipped 1 previous similar message [ 3239.391173] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3239.460972] Lustre: server umount lustre-MDT0000 complete [ 3241.105426] LustreError: 208828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786937550 with bad export cookie 8670321131160855109 [ 3241.112873] LustreError: 208828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3241.113382] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3241.245835] Lustre: server umount lustre-MDT0001 complete [ 3245.329144] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 3246.584655] Key type lgssc unregistered [ 3246.705476] LNet: 212553:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3246.707989] LNetError: 212553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3246.717386] LNet: Removed LNI 192.168.202.134@tcp [ 3247.053063] Key type .llcrypt unregistered [ 3247.054414] Key type ._llcrypt unregistered [ 3255.180580] Key type ._llcrypt registered [ 3255.181600] Key type .llcrypt registered [ 3255.214214] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 3255.602223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3255.609569] alg: No test for adler32 (adler32-zlib) [ 3256.456061] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 3256.546932] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 3258.127112] Key type lgssc registered [ 3258.438794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3261.043911] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3261.051433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3262.190089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3263.615816] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3266.943723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3267.128595] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3268.544223] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3271.035601] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3272.895644] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3275.334050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3275.466256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3277.433985] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3280.142376] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3287.780085] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 3297.248498] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3297.253050] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3297.260870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3298.274630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3298.275780] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3298.283137] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3300.238874] Lustre: server umount lustre-OST0000 complete [ 3302.880118] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3302.884469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3302.891836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3302.894951] Lustre: Skipped 1 previous similar message [ 3307.766394] Lustre: server umount lustre-MDT0000 complete [ 3308.512654] LustreError: 213802:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3308.517479] LustreError: 213802:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3309.209372] LustreError: 216305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786937618 with bad export cookie 7080800132242464508 [ 3309.212379] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3309.212743] LustreError: 216305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3309.321292] Lustre: server umount lustre-MDT0001 complete [ 3313.069823] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 3314.145476] Key type lgssc unregistered [ 3314.273468] LNet: 217511:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3314.276878] LNetError: 217511:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3314.283435] LNet: Removed LNI 192.168.202.134@tcp [ 3314.648562] Key type .llcrypt unregistered [ 3314.650216] Key type ._llcrypt unregistered [ 3322.669778] Key type ._llcrypt registered [ 3322.670888] Key type .llcrypt registered [ 3322.717751] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 3323.053756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3323.119697] alg: No test for adler32 (adler32-zlib) [ 3323.969689] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 3324.049727] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 3325.631172] Key type lgssc registered [ 3326.143352] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3329.377333] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3329.390476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3330.562456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3332.339745] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3336.486276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3336.652288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3338.692662] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3342.074759] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3344.157300] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3347.295829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3347.425452] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3349.382514] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3352.828919] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3355.629608] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 3362.332259] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3362.371818] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3362.490643] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3365.215748] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3368.537483] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3369.343915] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3380.062992] Lustre: Permanently deactivating lustre-OST0001 [ 3380.073214] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3386.892465] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3391.401834] Lustre: Permanently reactivating lustre-OST0001 [ 3391.414147] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 3391.419045] Lustre: Skipped 2 previous similar messages [ 3396.628081] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3396.636690] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3396.641178] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3396.649271] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3396.671331] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3401.698070] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3401.699955] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3401.701307] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3401.702868] Lustre: Skipped 2 previous similar messages [ 3406.821088] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3406.822759] Lustre: Skipped 2 previous similar messages [ 3407.034300] Lustre: server umount lustre-OST0001 complete [ 3411.937327] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3411.937796] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3411.939335] LustreError: 222307:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3411.939348] LustreError: 222307:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3411.939902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3411.940407] LustreError: Skipped 1 previous similar message [ 3411.943295] Lustre: Skipped 1 previous similar message [ 3414.280402] Lustre: server umount lustre-OST0000 complete [ 3417.056745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3417.057544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3417.059862] Lustre: Skipped 1 previous similar message [ 3417.061686] Lustre: Skipped 1 previous similar message [ 3421.536199] Lustre: server umount lustre-MDT0000 complete [ 3422.177375] LustreError: 218752:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3422.182299] LustreError: 218752:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3422.757291] LustreError: 218733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786937732 with bad export cookie 17115763044261857016 [ 3422.761663] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3422.881684] Lustre: server umount lustre-MDT0001 complete [ 3426.486304] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 3427.535766] Key type lgssc unregistered [ 3427.662448] LNet: 223796:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3427.664601] LNetError: 223796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3427.674386] LNet: Removed LNI 192.168.202.134@tcp [ 3427.993579] Key type .llcrypt unregistered [ 3427.994619] Key type ._llcrypt unregistered [ 3441.948418] Key type ._llcrypt registered [ 3441.949330] Key type .llcrypt registered [ 3441.983883] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 3442.416518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3442.425955] alg: No test for adler32 (adler32-zlib) [ 3443.278882] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 3443.377673] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 3444.959238] Key type lgssc registered [ 3445.294555] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3448.346846] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3448.353862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3449.441380] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3449.447362] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3449.498181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3450.885765] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3454.604736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3454.633254] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3454.642371] Lustre: 225978:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3454.728586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3455.943360] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3458.978095] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3461.227354] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3464.106123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3464.192914] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3464.261420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3465.647631] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 3466.748764] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3469.317259] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3471.001457] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 3479.697636] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 3489.256572] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 3496.963446] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3512.272847] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3512.275940] Lustre: Skipped 1 previous similar message [ 3529.759297] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 3529.762416] Lustre: Skipped 1 previous similar message [ 3567.112794] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3567.115409] Lustre: Skipped 4 previous similar messages [ 3633.803706] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 3633.805666] Lustre: Skipped 8 previous similar messages [ 3639.778666] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3639.779779] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3639.780409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3639.783469] LustreError: Skipped 1 previous similar message [ 3639.786650] Lustre: Skipped 1 previous similar message [ 3641.251879] Lustre: server umount lustre-OST0000 complete [ 3644.384120] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3644.386775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3644.391708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3644.394465] Lustre: Skipped 1 previous similar message [ 3650.016072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3650.017656] Lustre: Skipped 2 previous similar messages [ 3655.135943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3655.137370] Lustre: Skipped 1 previous similar message [ 3657.183179] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3657.269921] Lustre: server umount lustre-MDT0000 complete [ 3658.797966] LustreError: 225968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786937968 with bad export cookie 7684374533899502042 [ 3658.800630] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3658.801097] LustreError: 225968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3658.911515] Lustre: server umount lustre-MDT0001 complete [ 3662.716678] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 3663.832956] Key type lgssc unregistered [ 3663.951441] LNet: 230778:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3663.955986] LNetError: 230778:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3663.965412] LNet: Removed LNI 192.168.202.134@tcp [ 3664.402680] Key type .llcrypt unregistered [ 3664.403741] Key type ._llcrypt unregistered [ 3673.012547] Key type ._llcrypt registered [ 3673.013637] Key type .llcrypt registered [ 3673.055425] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 3673.475934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3673.483528] alg: No test for adler32 (adler32-zlib) [ 3674.320749] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 3674.404785] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 3675.983149] Key type lgssc registered [ 3676.462502] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3678.991248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3678.996420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3680.126780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3681.497192] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3684.943966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3685.066117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3686.690256] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3689.664340] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3691.707306] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3694.603211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3694.773940] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3697.170730] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3700.122756] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3705.005193] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 3716.575954] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3716.578202] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3716.582012] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3717.605141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3717.609846] Lustre: Skipped 2 previous similar messages [ 3722.139536] Lustre: server umount lustre-OST0000 complete [ 3726.304539] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3726.311945] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3726.323787] Lustre: Skipped 1 previous similar message [ 3726.328626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3728.864775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3728.865394] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3733.984313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3733.986237] Lustre: Skipped 2 previous similar messages [ 3738.079249] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3738.166613] Lustre: server umount lustre-MDT0000 complete [ 3739.105391] LustreError: 232166:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3739.111041] LustreError: 232166:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3739.814926] LustreError: 234743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786938049 with bad export cookie 15526171322865659355 [ 3739.822055] LustreError: 234743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3739.822509] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3739.963656] Lustre: server umount lustre-MDT0001 complete [ 3743.594236] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 3744.720442] Key type lgssc unregistered [ 3744.884627] LNet: 235861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3744.888502] LNetError: 235861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3744.901451] LNet: Removed LNI 192.168.202.134@tcp [ 3745.327695] Key type .llcrypt unregistered [ 3745.330069] Key type ._llcrypt unregistered [ 3754.783472] Key type ._llcrypt registered [ 3754.784687] Key type .llcrypt registered [ 3754.826129] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 3755.307348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3755.325806] alg: No test for adler32 (adler32-zlib) [ 3756.189089] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 3756.285824] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 3757.871160] Key type lgssc registered [ 3758.434827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4022.508311] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 4023.583075] Key type lgssc unregistered [ 4023.724577] LNet: 237873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4023.729624] LNetError: 237873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4023.743549] LNet: Removed LNI 192.168.202.134@tcp [ 4024.218582] Key type .llcrypt unregistered [ 4024.220396] Key type ._llcrypt unregistered [ 4032.987529] Key type ._llcrypt registered [ 4032.989401] Key type .llcrypt registered [ 4033.026762] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 4040.650670] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 4040.973790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4041.008223] alg: No test for adler32 (adler32-zlib) [ 4041.857220] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 4041.942359] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 4043.519174] Key type lgssc registered [ 4044.061873] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4057.201095] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 3936 sec ========== 23:46:05 (1786938365) [ 4058.006978] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:46:05 (1786938365) === [ 4059.381574] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:46:07 (1786938367) === [ 4075.228904] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 4076.614593] Key type lgssc unregistered [ 4076.798666] LNet: 243082:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4076.805627] LNetError: 243082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4076.818532] LNet: Removed LNI 192.168.202.134@tcp [ 4077.262185] Key type .llcrypt unregistered [ 4077.264307] Key type ._llcrypt unregistered