[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 392722564 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002187] x2apic enabled [ 0.003004] Switched APIC routing to physical x2apic. [ 0.003940] kvm-guest: setup PV IPIs [ 0.005882] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.006009] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.007004] pid_max: default: 32768 minimum: 301 [ 0.008079] LSM: Security Framework initializing [ 0.009025] Yama: becoming mindful. [ 0.010018] SELinux: Initializing. [ 0.011037] *** VALIDATE selinux *** [ 0.017561] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.021144] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.022104] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.023072] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024074] *** VALIDATE tmpfs *** [ 0.025142] *** VALIDATE proc *** [ 0.025968] *** VALIDATE cgroup *** [ 0.026004] *** VALIDATE cgroup2 *** [ 0.027184] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.028103] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.029003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.030018] Spectre V2 : User space: Vulnerable [ 0.031003] Speculative Store Bypass: Vulnerable [ 0.033520] debug: unmapping init [mem 0xffffffffb5259000-0xffffffffb5260fff] [ 0.035881] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.036402] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.037009] ... version: 2 [ 0.037816] ... bit width: 48 [ 0.038005] ... generic registers: 4 [ 0.038762] ... value mask: 0000ffffffffffff [ 0.039005] ... max period: 00007fffffffffff [ 0.040005] ... fixed-purpose events: 3 [ 0.040768] ... event mask: 000000070000000f [ 0.041111] rcu: Hierarchical SRCU implementation. [ 0.043058] smp: Bringing up secondary CPUs ... [ 0.044396] x86: Booting SMP configuration: [ 0.045014] .... node #0, CPUs: #1 #2 #3 [ 0.047311] smp: Brought up 1 node, 4 CPUs [ 0.048832] smpboot: Max logical packages: 1 [ 0.049006] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.127944] node 0 deferred pages initialised in 78ms [ 0.133237] devtmpfs: initialized [ 0.134303] x86/mm: Memory block size: 128MB [ 0.136314] gcov: version magic: 0x41383552 [ 0.138146] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.140079] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.143327] pinctrl core: initialized pinctrl subsystem [ 0.145154] [ 0.145696] ************************************************************* [ 0.148008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.150008] ** ** [ 0.153008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.155007] ** ** [ 0.157008] ** This means that this kernel is built to expose internal ** [ 0.160008] ** IOMMU data structures, which may compromise security on ** [ 0.162006] ** your system. ** [ 0.164006] ** ** [ 0.167009] ** If you see this message and you are not debugging the ** [ 0.169008] ** kernel, report this immediately to your vendor! ** [ 0.171011] ** ** [ 0.174008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.176009] ************************************************************* [ 0.179065] NET: Registered protocol family 16 [ 0.180303] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.182028] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.184048] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.186112] cpuidle: using governor menu [ 0.187307] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.188352] PCI: Using configuration type 1 for base access [ 0.190092] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.195112] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.196009] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.197100] cryptd: max_cpu_qlen set to 1000 [ 0.198150] ACPI: Added _OSI(Module Device) [ 0.199009] ACPI: Added _OSI(Processor Device) [ 0.199935] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.200007] ACPI: Added _OSI(Processor Aggregator Device) [ 0.204563] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.209044] ACPI: Interpreter enabled [ 0.209828] ACPI: PM: (supports S0 S3 S4 S5) [ 0.210006] ACPI: Using IOAPIC for interrupt routing [ 0.210987] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.213226] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.219623] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.221017] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.222007] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.224029] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.226837] acpiphp: Slot [2] registered [ 0.227070] acpiphp: Slot [5] registered [ 0.227849] acpiphp: Slot [6] registered [ 0.229050] acpiphp: Slot [7] registered [ 0.229867] acpiphp: Slot [8] registered [ 0.230049] acpiphp: Slot [9] registered [ 0.230826] acpiphp: Slot [10] registered [ 0.232044] acpiphp: Slot [3] registered [ 0.232835] acpiphp: Slot [4] registered [ 0.234043] acpiphp: Slot [11] registered [ 0.234828] acpiphp: Slot [12] registered [ 0.235042] acpiphp: Slot [13] registered [ 0.235804] acpiphp: Slot [14] registered [ 0.237043] acpiphp: Slot [15] registered [ 0.237858] acpiphp: Slot [16] registered [ 0.238049] acpiphp: Slot [17] registered [ 0.238835] acpiphp: Slot [18] registered [ 0.240078] acpiphp: Slot [19] registered [ 0.240858] acpiphp: Slot [20] registered [ 0.241042] acpiphp: Slot [21] registered [ 0.241885] acpiphp: Slot [22] registered [ 0.243058] acpiphp: Slot [23] registered [ 0.244105] acpiphp: Slot [24] registered [ 0.245068] acpiphp: Slot [25] registered [ 0.245999] acpiphp: Slot [26] registered [ 0.247050] acpiphp: Slot [27] registered [ 0.247883] acpiphp: Slot [28] registered [ 0.248066] acpiphp: Slot [29] registered [ 0.248962] acpiphp: Slot [30] registered [ 0.250046] acpiphp: Slot [31] registered [ 0.250987] PCI host bridge to bus 0000:00 [ 0.252011] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.253008] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.255008] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.256008] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.258009] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.260021] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.261114] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.262764] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.264813] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.272009] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.275061] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.276007] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.278010] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.280008] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.281482] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.283571] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.285035] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.286481] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.290830] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.298992] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.303007] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.307097] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.316009] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.320008] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.332010] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.337000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.348009] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.356009] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.371013] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.377010] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.380997] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.384936] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.395009] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.399784] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.402860] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.406683] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.417008] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.424859] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.428737] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.431764] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.442793] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.450393] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.457009] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.461008] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.474008] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.481484] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.483237] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.484215] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.485195] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.487234] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.491029] iommu: Default domain type: Passthrough [ 0.492294] SCSI subsystem initialized [ 0.493073] ACPI: bus type USB registered [ 0.493937] usbcore: registered new interface driver usbfs [ 0.495055] usbcore: registered new interface driver hub [ 0.496037] usbcore: registered new device driver usb [ 0.497126] pps_core: LinuxPPS API ver. 1 registered [ 0.499006] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.500017] PTP clock support registered [ 0.502048] EDAC MC: Ver: 3.0.0 [ 0.503365] PCI: Using ACPI for IRQ routing [ 0.504615] NetLabel: Initializing [ 0.505009] NetLabel: domain hash size = 128 [ 0.506006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.507063] NetLabel: unlabeled traffic allowed by default [ 0.508080] vgaarb: loaded [ 0.509249] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.510009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.516011] clocksource: Switched to clocksource kvm-clock [ 0.593293] VFS: Disk quotas dquot_6.6.0 [ 0.594276] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.595832] *** VALIDATE ramfs *** [ 0.596558] *** VALIDATE hugetlbfs *** [ 0.597523] pnp: PnP ACPI init [ 0.599185] pnp: PnP ACPI: found 6 devices [ 0.611546] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.613572] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.614931] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.616258] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.617684] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.619218] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.620906] NET: Registered protocol family 2 [ 0.622529] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.625733] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.627952] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.631610] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.633663] TCP: Hash tables configured (established 65536 bind 65536) [ 0.635604] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.637536] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.639328] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.641126] NET: Registered protocol family 1 [ 0.642818] RPC: Registered named UNIX socket transport module. [ 0.644074] RPC: Registered udp transport module. [ 0.645021] RPC: Registered tcp transport module. [ 0.646059] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.647431] NET: Registered protocol family 44 [ 0.648385] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.649668] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.650917] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.652265] PCI: CLS 0 bytes, default 64 [ 0.653267] Unpacking initramfs... [ 1.827302] debug: unmapping init [mem 0xffff9df93cc54000-0xffff9df93ffbffff] [ 1.831928] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.833355] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 1.836494] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.332158] Initialise system trusted keyrings [ 2.333652] Key type blacklist registered [ 2.335469] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.344624] zbud: loaded [ 2.347896] *** VALIDATE nfs *** [ 2.349143] *** VALIDATE nfs4 *** [ 2.350240] pstore: using deflate compression [ 2.352668] Platform Keyring initialized [ 2.429944] NET: Registered protocol family 38 [ 2.431312] Key type asymmetric registered [ 2.433134] Asymmetric key parser 'x509' registered [ 2.434656] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.438214] io scheduler mq-deadline registered [ 2.440146] io scheduler kyber registered [ 2.441381] io scheduler bfq registered [ 2.442661] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.444519] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.447128] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.449960] ACPI: Power Button [PWRF] [ 2.454556] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.461358] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.472083] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.477734] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.487872] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.515707] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.541596] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.545325] Non-volatile memory driver v1.3 [ 2.546485] Linux agpgart interface v0.103 [ 2.569972] virtio_blk virtio1: [vda] 145184 512-byte logical blocks (74.3 MB/70.9 MiB) [ 2.573263] vda: detected capacity change from 0 to 74334208 [ 2.585990] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.589295] vdb: detected capacity change from 0 to 1073741824 [ 2.602686] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.605917] vdc: detected capacity change from 0 to 2621440000 [ 2.619082] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.622249] vdd: detected capacity change from 0 to 2621440000 [ 2.635181] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.637376] vde: detected capacity change from 0 to 4294967296 [ 2.648352] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.650312] vdf: detected capacity change from 0 to 4294967296 [ 2.655746] libphy: Fixed MDIO Bus: probed [ 2.659336] usbcore: registered new interface driver usbserial_generic [ 2.661154] usbserial: USB Serial support registered for generic [ 2.663281] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.666825] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.668362] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.671118] mousedev: PS/2 mouse device common for all mice [ 2.673578] rtc_cmos 00:05: RTC can wake from S4 [ 2.675449] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.676754] rtc_cmos 00:05: registered as rtc0 [ 2.680543] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.680744] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.684365] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.686172] intel_pstate: CPU model not supported [ 2.692946] hid: raw HID events driver (C) Jiri Kosina [ 2.694894] usbcore: registered new interface driver usbhid [ 2.696829] usbhid: USB HID core driver [ 2.698281] drop_monitor: Initializing network drop monitor service [ 2.700766] Initializing XFRM netlink socket [ 2.702681] NET: Registered protocol family 10 [ 2.705514] Segment Routing with IPv6 [ 2.706934] NET: Registered protocol family 17 [ 2.708724] mpls_gso: MPLS GSO support [ 2.713935] RAS: Correctable Errors collector initialized. [ 2.716114] AVX version of gcm_enc/dec engaged. [ 2.717737] AES CTR mode by8 optimization enabled [ 2.786111] sched_clock: Marking stable (2786083535, 0)->(3381386413, -595302878) [ 2.788711] registered taskstats version 1 [ 2.790404] Loading compiled-in X.509 certificates [ 2.791616] zswap: loaded using pool lzo/zbud [ 2.812739] Key type big_key registered [ 2.825633] Key type encrypted registered [ 2.827407] ima: No TPM chip found, activating TPM-bypass! [ 2.829495] ima: Allocated hash algorithm: sha1 [ 2.831213] ima: No architecture policies found [ 2.832958] evm: Initialising EVM extended attributes: [ 2.834971] evm: security.selinux [ 2.836270] evm: security.ima [ 2.837490] evm: security.capability [ 2.838871] evm: HMAC attrs: 0x1 [ 2.841484] rtc_cmos 00:05: setting system clock to 2026-07-08 13:20:05 UTC (1783516805) [ 2.848547] debug: unmapping init [mem 0xffffffffb6203000-0xffffffffb63fffff] [ 2.851808] debug: unmapping init [mem 0xffffffffb4f82000-0xffffffffb5258fff] [ 2.860080] Write protecting the kernel read-only data: 28672k [ 2.863452] debug: unmapping init [mem 0xffffffffb3603000-0xffffffffb37fffff] [ 2.865928] debug: unmapping init [mem 0xffffffffb3f14000-0xffffffffb3ffffff] [ 2.897599] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 2.907152] systemd[1]: Detected virtualization kvm. [ 2.908973] systemd[1]: Detected architecture x86-64. [ 2.910885] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 2.936439] systemd[1]: No hostname configured. [ 2.938307] systemd[1]: Set hostname to . [ 2.940284] random: systemd: uninitialized urandom read (16 bytes read) [ 2.942754] systemd[1]: Initializing machine ID from random generator. [ 2.987463] random: ln: uninitialized urandom read (6 bytes read) [ 3.060243] random: systemd: uninitialized urandom read (16 bytes read) [ 3.061593] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.065967] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.070847] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. Starting Journal Service... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.600320] device-mapper: uevent: version 1.0.3 [ 3.602640] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.366315] random: fast init done [ 4.413062] virtio_net virtio0 ens2: renamed from eth0 [ 4.568250] scsi host0: ata_piix [ 4.602595] scsi host1: ata_piix [ 4.604120] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.606591] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.373146] random: crng init done [ 9.376061] random: 7 urandom warning(s) missed due to ratelimiting [ 9.432401] dracut-initqueue[588]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.204243] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.450112] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.696281] SELinux: Disabled at runtime. [ 11.757387] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.765473] systemd[1]: Detected virtualization kvm. [ 11.767300] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.313126] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.316203] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.321397] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.325270] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.328192] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.336649] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.341139] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Created slice User and Session Slice. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Paths. Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control So[ 12.584868] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS cket. Starting udev Coldplug all Devices... [ OK ] Reached target Slices. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.064983] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.545620] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.547102] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.859308] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.908394] EDAC sbridge: Ver: 1.1.2 [ 15.872620] Key type dns_resolver registered [ 16.234905] NFS: Registering the id_resolver key type [ 16.237341] Key type id_resolver registered [ 16.239271] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Mark the need to relabel after reboot. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg357-server login: [ 41.206486] libcfs: loading out-of-tree module taints kernel. [ 41.222659] Key type ._llcrypt registered [ 41.223592] Key type .llcrypt registered [ 41.260820] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [ 48.332037] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 48.830941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 48.837323] alg: No test for adler32 (adler32-zlib) [ 49.792948] Lustre: Lustre: Build Version: 2.17.54_105_g95cff68 [ 50.046662] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 51.647235] Key type lgssc registered [ 52.177589] Lustre: Echo OBD driver; http://www.lustre.org/ [ 57.662925] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 69.986598] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 73.634613] hrtimer: interrupt took 3006202 ns [ 75.487892] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 75.503369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 76.623788] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 76.641339] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 76.689986] Lustre: lustre-MDT0000: new disk, initializing [ 76.741320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 76.757565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 78.993977] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 85.899832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 85.941809] Lustre: 6477:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 85.961826] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 85.964655] Lustre: Skipped 1 previous similar message [ 86.005753] Lustre: lustre-MDT0001: new disk, initializing [ 86.029773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 86.041547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 86.046206] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 87.985836] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 90.862730] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 95.215673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 95.319669] Lustre: lustre-OST0000: new disk, initializing [ 95.322321] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 95.326258] Lustre: 8413:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 95.353146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 97.438019] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 98.218914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 98.221320] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 98.234546] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 102.733546] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 102.777603] Lustre: lustre-OST0001: new disk, initializing [ 102.780090] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 102.782122] Lustre: 9485:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 102.798481] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 104.561325] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 108.009674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 108.012839] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 108.025554] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 110.266314] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 114.843564] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 116.736406] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing check_logdir /tmp/testlogs/ [ 118.144123] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing yml_node [ 119.382130] Lustre: DEBUG MARKER: Client: 2.17.54.105 [ 120.120062] Lustre: DEBUG MARKER: MDS: 2.17.54.105 [ 120.868645] Lustre: DEBUG MARKER: OSS: 2.17.54.105 [ 121.357161] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Jul 8 09:22:03 EDT 2026 [ 126.532203] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 127.061648] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 138.719721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 138.720810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 138.723295] Lustre: Skipped 3 previous similar messages [ 138.725570] Lustre: Skipped 2 previous similar messages [ 143.839730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 143.843047] Lustre: Skipped 4 previous similar messages [ 144.862737] Lustre: server umount lustre-MDT0000 complete [ 146.095796] LustreError: 6472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783516948 with bad export cookie 3457361879042568497 [ 146.097235] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 146.100571] LustreError: 6472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 148.959830] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 148.960164] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 148.966372] Lustre: Skipped 1 previous similar message [ 151.967596] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 151.969671] Lustre: Skipped 2 previous similar messages [ 152.287871] Lustre: server umount lustre-MDT0001 complete [ 159.670581] Lustre: server umount lustre-OST0000 complete [ 167.109512] Lustre: server umount lustre-OST0001 complete [ 169.197941] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [ 172.241388] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 183.237986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 183.314066] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 183.323927] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 183.355866] Lustre: lustre-MDT0000: new disk, initializing [ 183.374420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 183.379603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 184.618111] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 188.557319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 188.597637] Lustre: 15023:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 188.697370] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 189.904284] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 193.384682] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 195.005316] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 197.020314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 197.113953] Lustre: lustre-OST0000: new disk, initializing [ 197.115869] Lustre: Skipped 1 previous similar message [ 197.117772] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 197.119421] Lustre: Skipped 2 previous similar messages [ 197.121879] Lustre: 16993:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 197.149403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 197.152716] Lustre: Skipped 1 previous similar message [ 198.776579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 198.779013] Lustre: Skipped 1 previous similar message [ 198.780843] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 198.812063] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 198.911723] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 202.397917] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 203.764133] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 203.825890] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 205.066392] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 205.132953] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 207.757426] Lustre: server umount lustre-OST0000 complete [ 209.377256] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 209.377908] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 209.378156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 209.382357] Lustre: Skipped 1 previous similar message [ 219.616074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 219.618269] Lustre: Skipped 4 previous similar messages [ 223.199305] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 223.253307] Lustre: server umount lustre-MDT0000 complete [ 224.339108] LustreError: 15017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783517026 with bad export cookie 3457361879042570030 [ 224.342889] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 224.342980] LustreError: 15017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 235.032227] Lustre: server umount lustre-MDT0001 complete [ 236.149500] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 09:23:58 (1783517038) [ 238.396793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 238.539626] LustreError: 19256:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 238.546322] LustreError: 19256:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 238.562558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 239.851259] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 242.701488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 244.342626] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 247.580860] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 250.305451] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 253.490648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 256.499902] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 260.322930] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 273.376543] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 273.381738] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 273.391314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 274.400029] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 279.313722] Lustre: server umount lustre-OST0000 complete [ 281.568811] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 281.575021] Lustre: Skipped 2 previous similar messages [ 286.751155] Lustre: server umount lustre-MDT0000 complete [ 288.181855] LustreError: 19238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783517090 with bad export cookie 3457361879042570912 [ 288.184336] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 288.186375] LustreError: 19238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 288.330362] Lustre: server umount lustre-MDT0001 complete [ 292.175753] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 293.271150] Key type lgssc unregistered [ 293.405781] LNet: 22805:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 293.409805] LNetError: 22805:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 293.424493] LNet: Removed LNI 192.168.203.157@tcp [ 293.801234] Key type .llcrypt unregistered [ 293.802664] Key type ._llcrypt unregistered [ 302.923392] Key type ._llcrypt registered [ 302.925223] Key type .llcrypt registered [ 302.966130] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 303.388746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 303.395939] alg: No test for adler32 (adler32-zlib) [ 304.258453] Lustre: Lustre: Build Version: 2.17.54_105_g95cff68 [ 304.357385] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 305.935165] Key type lgssc registered [ 306.321351] Lustre: Echo OBD driver; http://www.lustre.org/ [ 309.080280] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 309.088075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 310.261749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 311.771076] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 314.975901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 315.128172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 316.724169] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 319.762063] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 321.826801] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 324.398442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 324.552975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 326.719271] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 329.365102] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 334.823151] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 339.940106] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 339.940217] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 339.941458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 339.944897] Lustre: Skipped 1 previous similar message [ 344.916534] Lustre: server umount lustre-OST0000 complete [ 350.176869] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 350.177465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 350.181782] Lustre: Skipped 1 previous similar message [ 350.186279] Lustre: Skipped 2 previous similar messages [ 352.352327] Lustre: server umount lustre-MDT0000 complete [ 353.741515] LustreError: 24927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783517156 with bad export cookie 9174419152896530266 [ 353.744334] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 353.747121] LustreError: 24927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 353.878476] Lustre: server umount lustre-MDT0001 complete [ 357.853058] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 358.953273] Key type lgssc unregistered [ 359.080722] LNet: 28163:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 359.084453] LNetError: 28163:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 359.096365] LNet: Removed LNI 192.168.203.157@tcp [ 359.422142] Key type .llcrypt unregistered [ 359.423965] Key type ._llcrypt unregistered [ 368.943790] Key type ._llcrypt registered [ 368.945798] Key type .llcrypt registered [ 368.989984] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 369.314353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 369.329812] alg: No test for adler32 (adler32-zlib) [ 370.200271] Lustre: Lustre: Build Version: 2.17.54_105_g95cff68 [ 370.298986] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 371.887181] Key type lgssc registered [ 372.329288] Lustre: Echo OBD driver; http://www.lustre.org/ [ 375.221614] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 375.229330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 376.381263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 377.755740] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 380.653167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 380.799671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 382.174224] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 384.600853] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 386.472514] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 391.173553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 391.295061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 393.374494] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 396.034032] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 401.577262] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 411.106839] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 411.108772] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 411.109508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 411.114192] Lustre: Skipped 1 previous similar message [ 411.761562] Lustre: server umount lustre-OST0000 complete [ 413.500377] Lustre: server umount lustre-MDT0000 complete [ 415.143604] LustreError: 29543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783517217 with bad export cookie 6258747184255411198 [ 415.146867] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 415.149050] LustreError: 29543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 415.316166] Lustre: server umount lustre-MDT0001 complete [ 420.156301] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 421.433811] Key type lgssc unregistered [ 421.610440] LNet: 33456:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 421.614039] LNetError: 33456:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 421.625639] LNet: Removed LNI 192.168.203.157@tcp [ 421.981938] Key type .llcrypt unregistered [ 421.983619] Key type ._llcrypt unregistered [ 432.886750] Key type ._llcrypt registered [ 432.888267] Key type .llcrypt registered [ 432.931635] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 433.325852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 433.355082] alg: No test for adler32 (adler32-zlib) [ 434.235527] Lustre: Lustre: Build Version: 2.17.54_105_g95cff68 [ 434.339830] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 435.935182] Key type lgssc registered [ 436.473687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 439.913391] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 439.923929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 441.129711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 443.029816] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 446.925268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 447.084813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 448.716101] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 451.910738] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 454.255238] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 457.247540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 457.315714] LustreError: 36662:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 457.330217] LustreError: 36662:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 457.411539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 459.755969] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 462.553039] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 465.898245] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 473.056038] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 473.059359] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 473.065686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 476.130276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 476.132983] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 476.137199] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 477.978909] Lustre: server umount lustre-OST0000 complete [ 481.249161] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 481.250149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 481.255300] Lustre: Skipped 1 previous similar message [ 481.261745] Lustre: Skipped 2 previous similar messages [ 485.605401] Lustre: server umount lustre-MDT0000 complete [ 486.369090] LustreError: 34843:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 486.376974] LustreError: 34843:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 487.035197] LustreError: 34824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783517289 with bad export cookie 18121377512328905664 [ 487.038956] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 487.041058] LustreError: 34824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 487.163349] Lustre: server umount lustre-MDT0001 complete [ 491.282376] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 492.410574] Key type lgssc unregistered [ 492.545793] LNet: 38375:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 492.549073] LNetError: 38375:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 492.561857] LNet: Removed LNI 192.168.203.157@tcp [ 492.903321] Key type .llcrypt unregistered [ 492.905192] Key type ._llcrypt unregistered [ 502.022901] Key type ._llcrypt registered [ 502.024031] Key type .llcrypt registered [ 502.060526] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 502.387546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 502.415064] alg: No test for adler32 (adler32-zlib) [ 503.269289] Lustre: Lustre: Build Version: 2.17.54_105_g95cff68 [ 503.357180] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 504.943143] Key type lgssc registered [ 505.310897] Lustre: Echo OBD driver; http://www.lustre.org/ [ 508.120229] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 508.126777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 509.266334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 510.678618] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 513.594690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 513.740312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 515.241664] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 517.826243] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 519.793173] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 522.143565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 522.264540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 524.318742] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 526.740102] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 533.545677] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 539.616597] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 539.620146] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 539.625761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 542.456190] Lustre: lustre-OST0000: Not available for connect from 192.168.203.57@tcp (stopping) [ 543.713530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 543.715278] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 543.716655] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 545.678460] Lustre: server umount lustre-OST0000 complete [ 559.072289] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 559.073567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 559.076918] Lustre: Skipped 1 previous similar message [ 559.081199] Lustre: Skipped 2 previous similar messages [ 564.192151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 564.194960] Lustre: Skipped 1 previous similar message [ 564.954174] Lustre: server umount lustre-MDT0000 complete [ 566.253258] LustreError: 39744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783517368 with bad export cookie 7740032952483606728 [ 566.256342] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 566.257980] LustreError: 39744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 572.408837] Lustre: server umount lustre-MDT0001 complete [ 576.028502] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 577.100047] Key type lgssc unregistered [ 577.218332] LNet: 43489:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 577.221854] LNetError: 43489:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 577.231273] LNet: Removed LNI 192.168.203.157@tcp [ 577.545479] Key type .llcrypt unregistered [ 577.547296] Key type ._llcrypt unregistered [ 586.032722] Key type ._llcrypt registered [ 586.033864] Key type .llcrypt registered [ 586.070163] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 586.471348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 586.511841] alg: No test for adler32 (adler32-zlib) [ 587.368747] Lustre: Lustre: Build Version: 2.17.54_105_g95cff68 [ 587.455946] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 589.031107] Key type lgssc registered [ 589.373636] Lustre: Echo OBD driver; http://www.lustre.org/ [ 591.925737] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 591.933236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 593.073591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 594.417061] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 597.256696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 597.389098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 598.821243] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 601.241510] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 603.109284] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 605.412448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 605.550751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 607.533371] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 609.983377] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 617.830333] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 623.073477] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 623.074259] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 623.075027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 623.082567] Lustre: Skipped 2 previous similar messages [ 625.398778] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.57@tcp (stopping) [ 625.400718] Lustre: Skipped 2 previous similar messages [ 628.194412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 628.197592] Lustre: Skipped 2 previous similar messages [ 628.824545] Lustre: server umount lustre-MDT0000 complete [ 630.082429] LustreError: 44860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783517432 with bad export cookie 36923579789890903 [ 630.085072] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 630.086947] LustreError: 44860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 630.510410] LustreError: 44875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 630.517907] LustreError: 44875:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 630.522325] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.57@tcp (stopping) [ 633.311931] LustreError: 44876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 633.312315] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 635.360078] LustreError: 44875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 635.360365] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 635.368158] Lustre: Skipped 1 previous similar message [ 636.267809] Lustre: server umount lustre-MDT0001 complete [ 654.166660] Lustre: server umount lustre-OST0000 complete [ 659.365186] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 660.307322] Key type lgssc unregistered [ 660.423386] LNet: 48754:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 660.425790] LNetError: 48754:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 660.436292] LNet: Removed LNI 192.168.203.157@tcp [ 660.711371] Key type .llcrypt unregistered [ 660.712495] Key type ._llcrypt unregistered [ 669.786352] Key type ._llcrypt registered [ 669.787280] Key type .llcrypt registered [ 669.816857] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 670.136902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 670.174099] alg: No test for adler32 (adler32-zlib) [ 671.016690] Lustre: Lustre: Build Version: 2.17.54_105_g95cff68 [ 671.098477] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 672.679139] Key type lgssc registered [ 673.012821] Lustre: Echo OBD driver; http://www.lustre.org/ [ 675.408598] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 675.417035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 690.847455] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 695.967277] LustreError: 50117:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.157@tcp: failed processing log, type 4: rc = -110 [ 722.591245] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 724.433864] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 726.762641] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 888.933106] Lustre: server umount lustre-OST0000 complete [ 894.287069] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 895.291079] Key type lgssc unregistered [ 895.413464] LNet: 51789:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 895.416137] LNetError: 51789:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 895.422264] LNet: Removed LNI 192.168.203.157@tcp [ 895.707834] Key type .llcrypt unregistered [ 895.709921] Key type ._llcrypt unregistered [ 904.147405] Key type ._llcrypt registered [ 904.148499] Key type .llcrypt registered [ 904.185432] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 904.548952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 904.592391] alg: No test for adler32 (adler32-zlib) [ 905.455798] Lustre: Lustre: Build Version: 2.17.54_105_g95cff68 [ 905.550227] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 907.127163] Key type lgssc registered [ 907.487306] Lustre: Echo OBD driver; http://www.lustre.org/ [ 910.003685] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 910.009661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 911.153262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 912.462427] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 915.363449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 915.483442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 916.785268] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 919.018111] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 920.719389] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 922.849204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 922.975075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 924.898932] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 927.156945] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 931.298385] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 931.298711] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 931.304696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 933.962335] Lustre: server umount lustre-OST0000 complete [ 935.162921] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 935.163549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 935.166396] Lustre: Skipped 1 previous similar message [ 940.512087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 940.514228] Lustre: Skipped 2 previous similar messages [ 941.332289] Lustre: server umount lustre-MDT0000 complete [ 942.532303] LustreError: 53160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783517745 with bad export cookie 223867292803700563 [ 942.536555] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 942.537456] LustreError: 53160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 942.643329] Lustre: server umount lustre-MDT0001 complete [ 946.300371] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 947.239360] Key type lgssc unregistered [ 947.355296] LNet: 56706:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 947.357804] LNetError: 56706:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 947.365256] LNet: Removed LNI 192.168.203.157@tcp [ 947.653046] Key type .llcrypt unregistered [ 947.654272] Key type ._llcrypt unregistered [ 956.418953] Key type ._llcrypt registered [ 956.420657] Key type .llcrypt registered [ 956.460989] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 956.799527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 956.862082] alg: No test for adler32 (adler32-zlib) [ 957.720450] Lustre: Lustre: Build Version: 2.17.54_105_g95cff68 [ 957.811979] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 959.391135] Key type lgssc registered [ 959.759921] Lustre: Echo OBD driver; http://www.lustre.org/ [ 962.242305] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 962.248877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 977.695308] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 982.815345] LustreError: 58069:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.157@tcp: failed processing log, type 4: rc = -110 [ 1009.439215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1011.281599] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1013.537601] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1015.678520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1015.846727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1015.867885] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1017.074225] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1019.750606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1019.887300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1021.260538] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1023.515595] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1025.317875] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1026.015864] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1026.018468] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1026.022717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1029.092437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1029.092739] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1034.210773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1034.213627] Lustre: Skipped 2 previous similar messages [ 1039.330296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1039.332585] Lustre: Skipped 1 previous similar message [ 1040.351175] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1040.400284] Lustre: server umount lustre-OST0000 complete [ 1054.262073] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1054.263827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1054.265742] Lustre: Skipped 1 previous similar message [ 1054.270944] Lustre: Skipped 1 previous similar message [ 1060.501202] Lustre: server umount lustre-MDT0000 complete [ 1061.692210] LustreError: 58075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783517864 with bad export cookie 11750415916644050306 [ 1061.697825] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1067.918669] Lustre: server umount lustre-MDT0001 complete [ 1071.441856] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 1072.545225] Key type lgssc unregistered [ 1072.662454] LNet: 61808:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1072.664961] LNetError: 61808:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1072.675370] LNet: Removed LNI 192.168.203.157@tcp [ 1073.008954] Key type .llcrypt unregistered [ 1073.010136] Key type ._llcrypt unregistered [ 1082.364926] Key type ._llcrypt registered [ 1082.365932] Key type .llcrypt registered [ 1082.415503] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 1082.967469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1083.002664] alg: No test for adler32 (adler32-zlib) [ 1083.881977] Lustre: Lustre: Build Version: 2.17.54_105_g95cff68 [ 1083.986670] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 1085.583255] Key type lgssc registered [ 1086.071380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1088.864774] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1088.873228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1090.084759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1091.547872] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1094.409194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1094.546889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1095.988548] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1098.707770] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1100.842833] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1103.521756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1103.642536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1105.690860] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1108.105184] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1114.920990] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1125.348499] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1125.351173] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1125.353640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1125.357742] Lustre: Skipped 1 previous similar message [ 1125.362914] Lustre: Skipped 1 previous similar message [ 1126.742646] Lustre: server umount lustre-OST0000 complete [ 1130.464750] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1130.465537] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1130.466234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1130.467396] LustreError: Skipped 1 previous similar message [ 1130.473083] Lustre: Skipped 1 previous similar message [ 1134.573642] Lustre: server umount lustre-MDT0000 complete [ 1135.585077] LustreError: 63198:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1135.589763] LustreError: 63198:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1135.830364] LustreError: 65039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783517938 with bad export cookie 8681559268583785016 [ 1135.833447] LustreError: 65039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1135.833646] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1135.966783] Lustre: server umount lustre-MDT0001 complete [ 1139.668821] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 1140.743501] Key type lgssc unregistered [ 1140.873440] LNet: 66747:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1140.876920] LNetError: 66747:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1140.885292] LNet: Removed LNI 192.168.203.157@tcp [ 1141.231751] Key type .llcrypt unregistered [ 1141.232911] Key type ._llcrypt unregistered [ 1150.840563] Key type ._llcrypt registered [ 1150.842104] Key type .llcrypt registered [ 1150.876135] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 1151.174524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1151.186677] alg: No test for adler32 (adler32-zlib) [ 1152.033644] Lustre: Lustre: Build Version: 2.17.54_105_g95cff68 [ 1152.119758] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 1153.695285] Key type lgssc registered [ 1154.099484] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1156.874600] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 09:39:19 (1783517959) [ 1159.193366] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1159.199152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1160.379622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1161.766446] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1165.021824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1165.222295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1167.186439] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1170.436654] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1172.864483] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1175.838696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1176.019494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1178.559326] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1181.798173] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1187.306075] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1197.373040] Lustre: Failing over lustre-MDT0000 [ 1197.818126] Lustre: server umount lustre-MDT0000 complete [ 1198.339339] LustreError: 68421:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1198.362283] LustreError: 68421:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1199.585564] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1199.600718] Lustre: Skipped 1 previous similar message [ 1199.608287] LustreError: 69693:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1199.628924] LustreError: 69693:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1203.443913] LustreError: 68421:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1205.703723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1205.778563] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1205.843469] Lustre: *** cfs_fail_loc=135, val=0*** [ 1205.892503] LustreError: 71545:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 1205.897580] LustreError: 71545:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.157@tcp: cfg command failed: rc = -2 [ 1205.902017] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1205.902017] [ 1205.906578] LustreError: MGC192.168.203.157@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1205.912369] LustreError: 71533:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1205.915907] LustreError: 71533:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 1205.919593] LustreError: 71533:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 1205.923962] LustreError: 71533:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1205.946525] Lustre: server umount lustre-MDT0000 complete [ 1205.949516] LustreError: 71533:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1208.559147] LustreError: 68420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1208.567095] LustreError: 68420:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1208.570556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1208.612389] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1208.712401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1208.733876] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1210.051852] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1213.679866] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1213.927166] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1213.935474] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 1213.950176] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1214.943800] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1214.945551] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1214.948293] Lustre: Skipped 1 previous similar message [ 1214.950086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1219.043697] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1219.044974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1219.045182] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1219.053127] Lustre: Skipped 1 previous similar message [ 1220.946591] Lustre: server umount lustre-OST0000 complete [ 1224.160803] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1224.160975] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1224.165920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1224.169219] Lustre: Skipped 1 previous similar message [ 1228.306431] Lustre: server umount lustre-MDT0000 complete [ 1229.279932] LustreError: 68444:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1229.283766] LustreError: 68444:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1229.436145] LustreError: 68406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783518032 with bad export cookie 5720451222785210976 [ 1229.439269] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1229.440279] LustreError: 68406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1229.551858] Lustre: server umount lustre-MDT0001 complete [ 1233.063742] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 1234.113818] Key type lgssc unregistered [ 1234.237320] LNet: 73320:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1234.240419] LNetError: 73320:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1234.248351] LNet: Removed LNI 192.168.203.157@tcp [ 1234.569798] Key type .llcrypt unregistered [ 1234.571061] Key type ._llcrypt unregistered [ 1243.607732] Key type ._llcrypt registered [ 1243.609302] Key type .llcrypt registered [ 1243.655637] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 1244.216273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1244.229240] alg: No test for adler32 (adler32-zlib) [ 1245.098827] Lustre: Lustre: Build Version: 2.17.54_105_g95cff68 [ 1245.196611] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 1246.783214] Key type lgssc registered [ 1247.195246] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1249.981300] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1249.988113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1251.148727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1252.493298] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1255.258098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1255.382335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1256.675305] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1258.951451] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1260.703431] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1262.869022] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1262.986637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1264.715627] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1267.093386] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1276.326182] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1279.836449] Lustre: Failing over lustre-MDT0000 [ 1280.032680] Lustre: server umount lustre-MDT0000 complete [ 1280.992185] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1280.995138] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1281.001984] LustreError: 74709:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1281.006319] LustreError: 74709:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1281.504175] LustreError: 74728:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1281.505470] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1281.509627] LustreError: 74728:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1281.514434] Lustre: Skipped 1 previous similar message [ 1282.806714] LustreError: 76549:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1282.813115] LustreError: 76549:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1283.106788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1283.153582] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1283.211493] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1283.268134] LustreError: 77816:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 1283.270567] LustreError: 77816:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.157@tcp: cfg command failed: rc = -9 [ 1283.273534] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1283.273534] [ 1283.276357] LustreError: MGC192.168.203.157@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1283.280957] LustreError: 77804:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1283.284201] LustreError: 77804:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 1283.287223] LustreError: 77804:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 1283.289637] LustreError: 77804:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1283.300848] Lustre: server umount lustre-MDT0000 complete [ 1283.302260] LustreError: 77804:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1285.376351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1285.416533] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1285.506083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1285.528354] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1286.741096] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1287.917496] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1290.727086] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1290.736743] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1290.757317] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1291.743844] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1291.747140] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1291.752286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1295.841923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1295.843500] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1295.847903] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1297.817764] Lustre: server umount lustre-OST0000 complete [ 1300.961773] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1300.962314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1300.969165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1300.972424] Lustre: Skipped 2 previous similar messages [ 1305.303305] Lustre: server umount lustre-MDT0000 complete [ 1306.079993] LustreError: 74709:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1306.086080] LustreError: 74709:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1306.546077] LustreError: 74689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783518109 with bad export cookie 4188703705668491205 [ 1306.548679] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1306.549344] LustreError: 74689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1306.664486] Lustre: server umount lustre-MDT0001 complete [ 1310.109414] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 1311.162753] Key type lgssc unregistered [ 1311.284444] LNet: 79590:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1311.287822] LNetError: 79590:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1311.296375] LNet: Removed LNI 192.168.203.157@tcp [ 1311.620914] Key type .llcrypt unregistered [ 1311.622257] Key type ._llcrypt unregistered [ 1315.960652] Key type ._llcrypt registered [ 1315.962072] Key type .llcrypt registered [ 1315.999444] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 1316.093379] Key type .llcrypt unregistered [ 1316.094445] Key type ._llcrypt unregistered [ 1325.167631] Key type ._llcrypt registered [ 1325.169037] Key type .llcrypt registered [ 1325.209087] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 1325.695845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1325.703393] alg: No test for adler32 (adler32-zlib) [ 1326.564915] Lustre: Lustre: Build Version: 2.17.54_105_g95cff68 [ 1326.662984] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 1328.255167] Key type lgssc registered [ 1328.653934] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1331.350292] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1331.355984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1332.516680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1333.963219] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1336.996161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1337.138138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1338.570196] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1340.904980] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1342.628848] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1344.653076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1344.759567] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1346.524597] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1348.698642] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1359.079516] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1362.400027] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1362.402847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1362.409330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1364.449190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1364.451949] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1364.453936] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1367.952088] Lustre: server umount lustre-OST0000 complete [ 1369.568180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1369.568781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1369.571020] Lustre: Skipped 1 previous similar message [ 1369.572470] Lustre: Skipped 1 previous similar message [ 1374.687941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1374.689878] Lustre: Skipped 1 previous similar message [ 1379.808226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1379.810167] Lustre: Skipped 1 previous similar message [ 1383.391113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1383.454141] Lustre: server umount lustre-MDT0000 complete [ 1384.766253] LustreError: 83313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783518187 with bad export cookie 13145442479827949557 [ 1384.769939] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1384.770226] LustreError: 83313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1384.886614] Lustre: server umount lustre-MDT0001 complete [ 1388.517857] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 1389.564410] Key type lgssc unregistered [ 1389.694436] LNet: 85024:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1389.696938] LNetError: 85024:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1389.708295] LNet: Removed LNI 192.168.203.157@tcp [ 1390.044192] Key type .llcrypt unregistered [ 1390.045576] Key type ._llcrypt unregistered [ 1398.554353] Key type ._llcrypt registered [ 1398.555615] Key type .llcrypt registered [ 1398.594926] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 1398.906782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1398.939228] alg: No test for adler32 (adler32-zlib) [ 1399.800249] Lustre: Lustre: Build Version: 2.17.54_105_g95cff68 [ 1399.892082] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 1401.471176] Key type lgssc registered [ 1401.791498] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1404.257723] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1404.263556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1405.407150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1406.634753] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1409.299232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1409.415275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1410.612340] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1412.982906] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1414.718512] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1416.986131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1417.103556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1419.064746] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1421.417661] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1430.438350] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 1435.616443] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1435.617931] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1435.618299] LustreError: Skipped 1 previous similar message [ 1435.622249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1435.625086] Lustre: Skipped 1 previous similar message [ 1440.142665] Lustre: server umount lustre-OST0000 complete [ 1445.343786] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1445.346386] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1445.349308] Lustre: Skipped 1 previous similar message [ 1445.351079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1447.379299] Lustre: server umount lustre-MDT0000 complete [ 1448.468979] LustreError: 86393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783518251 with bad export cookie 6279345999299135397 [ 1448.471393] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1448.472336] LustreError: 86393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1448.575945] Lustre: server umount lustre-MDT0001 complete [ 1451.881310] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 1452.840880] Key type lgssc unregistered [ 1452.961346] LNet: 89962:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1452.963453] LNetError: 89962:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1452.974248] LNet: Removed LNI 192.168.203.157@tcp [ 1453.258152] Key type .llcrypt unregistered [ 1453.259851] Key type ._llcrypt unregistered [ 1462.025659] Key type ._llcrypt registered [ 1462.026729] Key type .llcrypt registered [ 1462.064850] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 1462.513828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1462.523497] alg: No test for adler32 (adler32-zlib) [ 1463.377923] Lustre: Lustre: Build Version: 2.17.54_105_g95cff68 [ 1463.466677] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 1465.055175] Key type lgssc registered [ 1465.470254] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1467.870222] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1467.875125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1469.013875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1470.464663] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1473.119894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1473.227788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1474.412414] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1476.596367] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1478.249147] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1480.368871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1480.481088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1482.316501] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1484.582095] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1493.801037] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 1499.108551] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1499.110352] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1499.112556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1499.116575] Lustre: Skipped 1 previous similar message [ 1499.119781] Lustre: Skipped 1 previous similar message [ 1504.225530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1504.228057] Lustre: Skipped 1 previous similar message [ 1504.592676] Lustre: server umount lustre-OST0000 complete [ 1509.345731] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1509.345895] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1509.347204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1509.351108] Lustre: Skipped 1 previous similar message [ 1514.464200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1514.466152] Lustre: Skipped 2 previous similar messages [ 1519.584091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1519.586237] Lustre: Skipped 1 previous similar message [ 1520.095075] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1520.156560] Lustre: server umount lustre-MDT0000 complete [ 1521.381074] LustreError: 91333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783518324 with bad export cookie 10598437503568198268 [ 1521.384553] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1521.385749] LustreError: 91333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1521.502955] Lustre: server umount lustre-MDT0001 complete [ 1525.204165] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 1526.271821] Key type lgssc unregistered [ 1526.394422] LNet: 94902:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1526.397763] LNetError: 94902:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1526.416517] LNet: Removed LNI 192.168.203.157@tcp [ 1526.768266] Key type .llcrypt unregistered [ 1526.769631] Key type ._llcrypt unregistered [ 1535.623802] Key type ._llcrypt registered [ 1535.624841] Key type .llcrypt registered [ 1535.665299] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 1535.983647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1535.992639] alg: No test for adler32 (adler32-zlib) [ 1536.837521] Lustre: Lustre: Build Version: 2.17.54_105_g95cff68 [ 1536.919374] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 1538.495131] Key type lgssc registered [ 1538.861311] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1541.541481] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1541.547904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1556.959660] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1562.079594] LustreError: 96279:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.157@tcp: failed processing log, type 4: rc = -110 [ 1588.703156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1590.533732] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1592.843559] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1594.718459] Lustre: server umount lustre-OST0000 complete [ 1597.481872] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 09:46:39 (1783518399) [ 1599.783028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1599.902735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1601.152990] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1603.833934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1603.953980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1605.202326] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1607.425213] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1609.013777] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1611.112016] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1611.212343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1612.994869] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1615.214522] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1624.552683] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 1629.668663] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1629.669860] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1629.675282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1629.676879] LustreError: Skipped 1 previous similar message [ 1629.681374] Lustre: Skipped 1 previous similar message [ 1631.315142] Lustre: server umount lustre-OST0000 complete [ 1634.710951] Lustre: server umount lustre-MDT0000 complete [ 1634.785390] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1634.785427] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1634.790610] LustreError: 97913:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1634.793425] Lustre: Skipped 2 previous similar messages [ 1634.798243] LustreError: 97913:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1636.006367] LustreError: 97896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783518438 with bad export cookie 7227904294453344006 [ 1636.008743] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1636.010128] LustreError: 97896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1636.143777] Lustre: server umount lustre-MDT0001 complete [ 1639.779411] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 1640.814827] Key type lgssc unregistered [ 1640.931334] LNet: 101620:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1640.933548] LNetError: 101620:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1640.942276] LNet: Removed LNI 192.168.203.157@tcp [ 1641.203132] Key type .llcrypt unregistered [ 1641.204401] Key type ._llcrypt unregistered [ 1655.933997] Key type ._llcrypt registered [ 1655.934939] Key type .llcrypt registered [ 1655.968228] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 1656.262520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1656.292466] alg: No test for adler32 (adler32-zlib) [ 1657.157175] Lustre: Lustre: Build Version: 2.17.54_105_g95cff68 [ 1657.240313] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 1658.823099] Key type lgssc registered [ 1659.177411] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1661.705538] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1661.711569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1662.879974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1664.162168] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1666.806142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1666.909668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1668.220290] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1670.499908] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1672.103126] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1674.167491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1674.273520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1675.996898] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1678.116144] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1687.587526] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 1692.642845] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1692.642919] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1692.644842] LustreError: Skipped 1 previous similar message [ 1692.645292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1692.647836] Lustre: Skipped 1 previous similar message [ 1692.651776] Lustre: Skipped 1 previous similar message [ 1693.518288] Lustre: server umount lustre-OST0000 complete [ 1696.720907] Lustre: server umount lustre-MDT0000 complete [ 1697.760460] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1697.760487] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1697.760957] LustreError: 103779:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1697.760963] LustreError: 103779:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1697.763409] Lustre: Skipped 1 previous similar message [ 1697.871549] LustreError: 103761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783518500 with bad export cookie 10467760559918250900 [ 1697.875202] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1697.876811] LustreError: 103761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1697.972827] Lustre: server umount lustre-MDT0001 complete [ 1701.511727] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 1702.489970] Key type lgssc unregistered [ 1702.615321] LNet: 107326:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1702.617597] LNetError: 107326:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1702.628268] LNet: Removed LNI 192.168.203.157@tcp [ 1702.937819] Key type .llcrypt unregistered [ 1702.938928] Key type ._llcrypt unregistered [ 1710.001370] Key type ._llcrypt registered [ 1710.002814] Key type .llcrypt registered [ 1710.041026] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 1710.445498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1710.453280] alg: No test for adler32 (adler32-zlib) [ 1711.313157] Lustre: Lustre: Build Version: 2.17.54_105_g95cff68 [ 1711.404534] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 1712.983156] Key type lgssc registered [ 1713.351937] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1715.994570] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1715.999759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1731.423381] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1736.543232] LustreError: 108546:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.157@tcp: failed processing log, type 4: rc = -110 [ 1763.167383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1765.150212] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1767.580719] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1769.841463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1769.942270] LustreError: MGC192.168.203.157@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1769.947046] LustreError: 109569:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1769.949714] LustreError: 109569:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 1769.952906] LustreError: 109569:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 1769.979359] Lustre: server umount lustre-MDT0000 complete [ 1769.980660] LustreError: 109569:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1782.786051] Lustre: server umount lustre-OST0000 complete [ 1785.575875] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [ 1788.548397] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 1799.341090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1799.411682] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1799.420236] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1799.450267] Lustre: lustre-MDT0000: new disk, initializing [ 1799.467576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1799.472756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1800.640413] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1804.511521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1804.552629] Lustre: 112355:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1804.574377] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1804.577505] Lustre: Skipped 1 previous similar message [ 1804.609174] Lustre: lustre-MDT0001: new disk, initializing [ 1804.638209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1804.648954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1804.652148] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1805.849499] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1809.408944] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1811.141096] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1813.234882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1813.328379] Lustre: lustre-OST0000: new disk, initializing [ 1813.330218] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1813.332326] Lustre: 114321:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1813.359966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1815.060783] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1815.096797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1815.101523] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1815.135794] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1818.439424] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1819.690325] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1819.750803] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1820.889542] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1820.954102] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1825.250359] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1825.250695] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1825.251251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1825.252744] LustreError: Skipped 1 previous similar message [ 1825.255984] Lustre: Skipped 1 previous similar message [ 1827.597302] Lustre: server umount lustre-OST0000 complete [ 1830.368654] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1830.369132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1830.369928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1830.369932] Lustre: Skipped 1 previous similar message [ 1830.379564] Lustre: Skipped 1 previous similar message [ 1835.488051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1835.490110] Lustre: Skipped 2 previous similar messages [ 1840.608241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1840.610731] Lustre: Skipped 1 previous similar message [ 1843.167142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1843.228362] Lustre: server umount lustre-MDT0000 complete [ 1844.535931] LustreError: 112347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783518647 with bad export cookie 1536727581962034643 [ 1844.538624] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1844.539187] LustreError: 112347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1844.653571] Lustre: server umount lustre-MDT0001 complete [ 1847.418329] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 09:50:49 (1783518649) [ 1848.425638] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 1853.276372] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [ 1856.222058] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 1867.091820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1867.169686] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1867.178419] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1867.210543] Lustre: lustre-MDT0000: new disk, initializing [ 1867.230514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1867.235728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1868.481429] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1872.603954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1872.643269] Lustre: 119214:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1872.664056] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1872.665562] Lustre: Skipped 1 previous similar message [ 1872.698878] Lustre: lustre-MDT0001: new disk, initializing [ 1872.738777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1872.742879] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1873.950194] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1877.459573] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1879.253201] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1881.301165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1881.389832] Lustre: lustre-OST0000: new disk, initializing [ 1881.391966] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1881.394678] Lustre: 121180:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1881.419552] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1881.422018] Lustre: Skipped 1 previous similar message [ 1883.362714] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1886.869363] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1888.217682] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1890.296075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1890.300188] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1890.329434] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1891.336409] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 1892.576607] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1892.642982] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1895.394523] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1895.394879] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1895.396063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1895.403829] Lustre: Skipped 1 previous similar message [ 1899.337724] Lustre: server umount lustre-OST0000 complete [ 1903.583927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1903.587837] LustreError: Skipped 1 previous similar message [ 1903.589137] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1903.592993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1903.595530] Lustre: Skipped 1 previous similar message [ 1906.715112] Lustre: server umount lustre-MDT0000 complete [ 1907.934204] LustreError: 119207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783518710 with bad export cookie 1536727581962035511 [ 1907.937426] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1907.938179] LustreError: 119207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1908.062772] Lustre: server umount lustre-MDT0001 complete [ 1910.898978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1911.068130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1912.441635] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1915.196338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1916.651167] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1919.162521] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1920.868796] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1922.855298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1924.604917] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1926.771304] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1939.517459] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 1940.447677] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1940.451274] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1940.454726] Lustre: Skipped 1 previous similar message [ 1940.456261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1940.457728] Lustre: Skipped 2 previous similar messages [ 1942.544348] Lustre: server umount lustre-OST0000 complete [ 1949.852667] Lustre: server umount lustre-MDT0000 complete [ 1951.131581] LustreError: 125255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783518753 with bad export cookie 1536727581962036393 [ 1951.135269] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1951.137146] LustreError: 125255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1951.263940] Lustre: server umount lustre-MDT0001 complete [ 1954.901680] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 1955.814437] Key type lgssc unregistered [ 1955.930331] LNet: 127060:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1955.932520] LNetError: 127060:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1955.941256] LNet: Removed LNI 192.168.203.157@tcp [ 1956.212302] Key type .llcrypt unregistered [ 1956.213243] Key type ._llcrypt unregistered [ 1961.753309] Key type ._llcrypt registered [ 1961.754969] Key type .llcrypt registered [ 1961.793288] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [ 1968.171775] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 1968.550736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1968.559284] alg: No test for adler32 (adler32-zlib) [ 1969.393984] Lustre: Lustre: Build Version: 2.17.54_105_g95cff68 [ 1969.476672] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 1971.055126] Key type lgssc registered [ 1971.366892] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1981.946626] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1981.952098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1983.026831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1983.036748] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1983.072282] Lustre: lustre-MDT0000: new disk, initializing [ 1983.094587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1983.100800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1984.405802] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1988.362674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1988.403864] Lustre: 130187:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1988.431260] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1988.434159] Lustre: Skipped 1 previous similar message [ 1988.470115] Lustre: lustre-MDT0001: new disk, initializing [ 1988.501959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1988.514157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1988.517739] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1989.780438] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1993.273702] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1994.934888] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1996.975979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1997.071720] Lustre: lustre-OST0000: new disk, initializing [ 1997.074085] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1997.077615] Lustre: 132151:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1997.102992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1998.868046] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2002.360278] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2003.726296] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2007.031262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2007.035198] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2007.064448] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2007.875992] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 2009.168857] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2009.235515] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2012.128271] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2012.128973] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2012.129969] LustreError: Skipped 1 previous similar message [ 2012.133808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2012.135830] Lustre: Skipped 1 previous similar message [ 2015.955550] Lustre: server umount lustre-OST0000 complete [ 2017.248145] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2017.248692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2017.251736] Lustre: Skipped 2 previous similar messages [ 2017.254266] Lustre: Skipped 1 previous similar message [ 2022.368049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2022.370481] Lustre: Skipped 1 previous similar message [ 2023.253954] Lustre: server umount lustre-MDT0000 complete [ 2024.375023] LustreError: 130981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783518827 with bad export cookie 1620833488778262503 [ 2024.378017] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2024.379431] LustreError: 130981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2024.485592] Lustre: server umount lustre-MDT0001 complete [ 2026.998336] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 09:53:49 (1783518829) [ 2028.990852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2029.139446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2030.298890] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2032.823947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2034.067313] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2036.352107] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2038.167124] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2042.976778] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2042.982876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2042.985476] Lustre: Skipped 1 previous similar message [ 2045.012907] Lustre: server umount lustre-MDT0000 complete [ 2046.244093] LustreError: 134610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783518848 with bad export cookie 1620833488778263371 [ 2046.247281] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2046.247441] LustreError: 134610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2046.355558] Lustre: server umount lustre-MDT0001 complete [ 2049.006177] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 09:54:11 (1783518851) [ 2050.994361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2066.399557] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2071.519438] LustreError: 136995:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.157@tcp: failed processing log, type 4: rc = -110 [ 2097.119270] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2097.121368] Lustre: Skipped 1 previous similar message [ 2098.854836] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2101.032645] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2101.683473] Lustre: server umount lustre-OST0000 complete [ 2104.080702] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 09:55:06 (1783518906) [ 2106.076172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2106.207061] LustreError: 138437:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2106.213440] LustreError: 138437:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2106.227376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2107.407501] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2109.947719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2111.291920] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2113.418843] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2115.017944] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2116.990166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2118.867647] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2121.283594] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2135.520274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2135.524682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2135.531588] Lustre: Skipped 1 previous similar message [ 2135.534527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2139.476943] Lustre: server umount lustre-MDT0000 complete [ 2140.744166] LustreError: 138419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783518943 with bad export cookie 1620833488778264008 [ 2140.747887] LustreError: 138419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2140.747987] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2140.865949] Lustre: server umount lustre-MDT0001 complete [ 2152.346335] Lustre: server umount lustre-OST0000 complete [ 2155.027646] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 09:55:57 (1783518957) [ 2157.226847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2157.375838] LustreError: 142106:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2157.398493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2157.400402] Lustre: Skipped 2 previous similar messages [ 2158.741589] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2161.582411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2163.038168] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2165.403152] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2167.165729] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2169.427881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2171.490949] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2173.874513] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2175.274485] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2182.883279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 2183.491595] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 2184.821049] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2184.904833] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2187.234262] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2187.236832] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2187.240611] Lustre: Skipped 2 previous similar messages [ 2187.242338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2187.245529] Lustre: Skipped 3 previous similar messages [ 2191.702416] Lustre: server umount lustre-OST0000 complete [ 2199.187036] Lustre: server umount lustre-MDT0000 complete [ 2200.416167] LustreError: 142088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783519003 with bad export cookie 1620833488778265065 [ 2200.419867] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2200.420012] LustreError: 142088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2200.549359] Lustre: server umount lustre-MDT0001 complete [ 2203.264467] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 09:56:45 (1783519005) [ 2205.472225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2220.959583] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2226.079507] LustreError: 146316:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.157@tcp: failed processing log, type 4: rc = -110 [ 2251.679435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2251.681446] Lustre: Skipped 2 previous similar messages [ 2253.668429] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2256.124690] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2258.373748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2258.511652] LustreError: 146341:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2258.561673] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 2259.864781] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2262.710763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2264.228245] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2266.638851] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2268.529037] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2270.002168] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2270.072506] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2271.493503] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2271.578659] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2272.225432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2272.226665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2272.228715] Lustre: Skipped 5 previous similar messages [ 2272.232517] Lustre: Skipped 8 previous similar messages [ 2278.356616] Lustre: server umount lustre-MDT0000 complete [ 2279.570074] LustreError: 146322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783519082 with bad export cookie 1620833488778266073 [ 2279.573708] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2279.576128] LustreError: 146322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2279.686232] Lustre: server umount lustre-MDT0001 complete [ 2291.163829] Lustre: server umount lustre-OST0000 complete [ 2293.843716] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 09:58:16 (1783519096) [ 2296.034233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2311.455542] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2316.575474] LustreError: 150507:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.157@tcp: failed processing log, type 4: rc = -110 [ 2342.175440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2342.178390] Lustre: Skipped 2 previous similar messages [ 2344.086359] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2346.428449] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2348.774590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2348.940770] LustreError: 150532:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2348.985965] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 2350.389545] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2353.223072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2354.766425] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2357.110265] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2358.937315] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2361.270208] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2361.350282] Lustre: lustre-OST0001: new disk, initializing [ 2361.352382] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2361.354994] Lustre: 153376:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2363.314254] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2367.037212] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2368.478540] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2370.552377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2370.555833] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2370.590582] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2371.604509] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 2372.949693] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2373.020501] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2374.112265] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2374.115561] LustreError: Skipped 1 previous similar message [ 2374.117250] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2374.122126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2374.124105] Lustre: Skipped 3 previous similar messages [ 2379.705969] Lustre: server umount lustre-OST0000 complete [ 2380.768641] LustreError: 151580:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2380.773204] LustreError: 151580:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2385.891163] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2385.894158] LustreError: Skipped 2 previous similar messages [ 2387.023228] Lustre: server umount lustre-OST0001 complete [ 2394.412266] Lustre: server umount lustre-MDT0000 complete [ 2395.675483] LustreError: 154937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783519198 with bad export cookie 1620833488778267025 [ 2395.678931] LustreError: 154937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2395.680131] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2395.807743] Lustre: server umount lustre-MDT0001 complete [ 2403.416044] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 10:00:05 (1783519205) [ 2404.000762] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 2404.554096] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 2405.091787] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 10:00:07 (1783519207) [ 2407.357762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2407.459257] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2407.472140] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2408.983244] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2411.909925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2411.939505] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2411.948182] Lustre: 156951:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2413.401096] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2415.798284] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2417.668594] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2419.963912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2420.034379] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2422.157781] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2423.823891] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 2424.698597] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2426.235619] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2426.328467] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2427.714392] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2427.792443] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2428.896734] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2428.898507] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2428.899790] LustreError: Skipped 2 previous similar messages [ 2428.904469] Lustre: Skipped 6 previous similar messages [ 2430.478020] Lustre: server umount lustre-OST0000 complete [ 2444.218958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2446.194055] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2448.420469] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2448.424267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2448.424341] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2448.428921] Lustre: Skipped 1 previous similar message [ 2448.484384] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2450.038676] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2450.113890] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2451.519845] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2451.598271] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2453.427281] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9cfc442a0000.ost_server_uuid 50 [ 2453.932140] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9cfc442a0000.ost_server_uuid in FULL state after 0 sec [ 2453.990988] LustreError: 156957:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.57@tcp) returned error from blocking AST (req@000000005e3f837d x1870154980054784 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000fbde5039/0x167e5b7872af92dd lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.57@tcp remote: 0xf8558cb38015d6fa expref: 5 pid: 156958 timeout: 2553 lvb_type: 0 lru_score: 0 lru_type: 0 [ 2454.009590] LustreError: lustre-MDT0000: A client on nid 192.168.203.57@tcp was evicted due to a lock blocking callback time out: rc -107 [ 2454.014201] LustreError: 156946:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 1s: evicting client at 192.168.203.57@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000fbde5039/0x167e5b7872af92dd lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.57@tcp remote: 0xf8558cb38015d6fa expref: 6 pid: 156958 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 2458.593429] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2458.594863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2458.595493] LustreError: Skipped 1 previous similar message [ 2458.597404] Lustre: Skipped 10 previous similar messages [ 2461.196792] Lustre: server umount lustre-OST0000 complete [ 2468.568651] Lustre: server umount lustre-MDT0000 complete [ 2468.833126] LustreError: 156961:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2468.840895] LustreError: 156961:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2469.783492] LustreError: 157690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783519272 with bad export cookie 1620833488778268208 [ 2469.787355] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2469.789513] LustreError: 157690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2484.191119] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2484.281723] Lustre: server umount lustre-MDT0001 complete [ 2487.819697] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 2488.823246] Key type lgssc unregistered [ 2488.949418] LNet: 163069:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2488.952504] LNetError: 163069:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2488.963328] LNet: Removed LNI 192.168.203.157@tcp [ 2489.254878] Key type .llcrypt unregistered [ 2489.256475] Key type ._llcrypt unregistered [ 2498.121545] Key type ._llcrypt registered [ 2498.122622] Key type .llcrypt registered [ 2498.160494] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 2498.585971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2498.591389] alg: No test for adler32 (adler32-zlib) [ 2499.441237] Lustre: Lustre: Build Version: 2.17.54_105_g95cff68 [ 2499.526642] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 2501.103164] Key type lgssc registered [ 2501.435348] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2503.890705] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2503.896708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2505.036163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2506.291445] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2509.007701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2509.125053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2510.431685] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2512.670486] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2514.345712] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2516.460643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2516.581512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2518.369815] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2520.671807] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2530.920164] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 2533.156769] Lustre: Failing over lustre-MDT0000 [ 2533.368639] Lustre: server umount lustre-MDT0000 complete [ 2534.880230] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2534.884219] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2534.891059] LustreError: 164458:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2534.898451] LustreError: 164458:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2536.416441] LustreError: 164457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2536.417456] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2541.536118] LustreError: 165937:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2541.541690] LustreError: 165937:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2544.309658] LustreError: 164439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.57@tcp arrived at 1783519346 with bad export cookie 9714350243253266149 [ 2544.317189] LustreError: 164439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2546.657316] LustreError: 164457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2546.663962] LustreError: 164457:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2546.728026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2546.772108] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2546.871623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2546.894149] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2548.189270] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2548.918539] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2548.921579] Lustre: lustre-MDT0000: Denying connection for new client 72eeb735-41c4-4c0b-a3bf-eeca39a5371e (at 192.168.203.57@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2552.295278] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2557.410102] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2557.411246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2557.416647] Lustre: Skipped 2 previous similar messages [ 2561.807309] Lustre: server umount lustre-OST0000 complete [ 2563.042401] LustreError: 168202:0:(ldlm_lib.c:2990:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2563.046192] Lustre: 167555:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2563.048954] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2563.055182] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2563.059167] LustreError: 167555:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000b99e3ccd x1870155535648896/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2563.064833] LustreError: 167555:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 2563.067922] LustreError: 167555:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 2563.071207] Lustre: lustre-MDT0000: Recovery over after 0:15, of 2 clients 0 recovered and 2 were evicted. [ 2563.172967] Lustre: server umount lustre-MDT0000 complete [ 2564.429412] LustreError: 164438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783519367 with bad export cookie 9714350243253266478 [ 2564.433329] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2564.434196] LustreError: 163768:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 2570.597466] Lustre: server umount lustre-MDT0001 complete [ 2573.971493] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 2574.981140] Key type lgssc unregistered [ 2575.107347] LNet: 168953:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2575.109766] LNetError: 168953:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2575.116282] LNet: Removed LNI 192.168.203.157@tcp [ 2575.430871] Key type .llcrypt unregistered [ 2575.432038] Key type ._llcrypt unregistered [ 2584.105733] Key type ._llcrypt registered [ 2584.107586] Key type .llcrypt registered [ 2584.146029] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 2584.581951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2584.591418] alg: No test for adler32 (adler32-zlib) [ 2585.452897] Lustre: Lustre: Build Version: 2.17.54_105_g95cff68 [ 2585.541031] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 2587.119178] Key type lgssc registered [ 2587.491952] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2590.055160] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2590.061857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2591.203167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2592.564611] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2595.403804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2595.527810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2596.895347] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2599.263267] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2601.006175] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2603.163909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2603.269371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2605.158679] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2607.449263] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2615.590792] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 2620.897940] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2620.899229] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2620.901099] LustreError: Skipped 1 previous similar message [ 2620.905773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2620.908924] Lustre: Skipped 1 previous similar message [ 2621.262486] Lustre: server umount lustre-OST0000 complete [ 2624.533378] Lustre: server umount lustre-MDT0000 complete [ 2625.778656] LustreError: 170326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783519428 with bad export cookie 9085082724172034300 [ 2625.781986] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2625.783689] LustreError: 170326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2625.910144] Lustre: server umount lustre-MDT0001 complete [ 2629.536494] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 2630.602221] Key type lgssc unregistered [ 2630.754510] LNet: 173894:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2630.758205] LNetError: 173894:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2630.768317] LNet: Removed LNI 192.168.203.157@tcp [ 2631.088080] Key type .llcrypt unregistered [ 2631.089293] Key type ._llcrypt unregistered [ 2642.497390] Key type ._llcrypt registered [ 2642.498414] Key type .llcrypt registered [ 2642.533338] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 2642.951788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2642.960818] alg: No test for adler32 (adler32-zlib) [ 2643.805717] Lustre: Lustre: Build Version: 2.17.54_105_g95cff68 [ 2643.896573] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 2645.471169] Key type lgssc registered [ 2645.849461] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2648.476996] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2648.484906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2649.629749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2650.956653] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2653.697180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2653.824621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2655.140967] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2657.458725] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2659.249030] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2661.478064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2661.597369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2663.503804] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2665.940542] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2674.915090] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 2679.264151] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2679.267188] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2679.273772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2680.289791] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2680.291016] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2680.291300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2683.920677] Lustre: server umount lustre-OST0000 complete [ 2685.408278] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2685.417130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2685.419119] Lustre: Skipped 2 previous similar messages [ 2690.528051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2690.530203] Lustre: Skipped 1 previous similar message [ 2691.291073] Lustre: server umount lustre-MDT0000 complete [ 2692.546369] LustreError: 175550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783519495 with bad export cookie 18191371587122361070 [ 2692.549343] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2692.550849] LustreError: 175550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2692.663752] Lustre: server umount lustre-MDT0001 complete [ 2696.496339] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 2697.558494] Key type lgssc unregistered [ 2697.694394] LNet: 179118:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2697.697529] LNetError: 179118:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2697.706364] LNet: Removed LNI 192.168.203.157@tcp [ 2698.045767] Key type .llcrypt unregistered [ 2698.047367] Key type ._llcrypt unregistered [ 2706.400197] Key type ._llcrypt registered [ 2706.401383] Key type .llcrypt registered [ 2706.432556] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 2706.787387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2706.792837] alg: No test for adler32 (adler32-zlib) [ 2707.632203] Lustre: Lustre: Build Version: 2.17.54_105_g95cff68 [ 2707.714970] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 2709.287116] Key type lgssc registered [ 2709.651049] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2713.263357] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2713.271435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2714.408262] Lustre: *** cfs_fail_loc=135, val=0*** [ 2714.452000] LustreError: 180630:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2714.456140] LustreError: 180630:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.157@tcp: cfg command failed: rc = -2 [ 2714.461312] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2714.461312] [ 2714.465360] LustreError: MGC192.168.203.157@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2714.471629] LustreError: 180592:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2714.474404] LustreError: 180592:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2714.477134] LustreError: 180592:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2714.479514] LustreError: 180592:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2714.512856] Lustre: server umount lustre-MDT0000 complete [ 2714.514559] LustreError: 180592:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2716.671293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2716.801538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2718.055744] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2720.739026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2720.847301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2722.141395] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2724.470562] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2726.203829] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2730.976649] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2730.976892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2730.977547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2730.984956] Lustre: Skipped 1 previous similar message [ 2736.096195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2736.098089] Lustre: Skipped 2 previous similar messages [ 2741.215110] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2741.216076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2741.219990] Lustre: Skipped 1 previous similar message [ 2741.270906] Lustre: server umount lustre-MDT0000 complete [ 2742.451050] LustreError: 180982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783519545 with bad export cookie 11022822343012152155 [ 2742.453723] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2742.455710] LustreError: 180982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2742.565119] Lustre: server umount lustre-MDT0001 complete [ 2745.861156] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 2746.800748] Key type lgssc unregistered [ 2746.924439] LNet: 183320:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2746.927129] LNetError: 183320:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2746.934278] LNet: Removed LNI 192.168.203.157@tcp [ 2747.230823] Key type .llcrypt unregistered [ 2747.231740] Key type ._llcrypt unregistered [ 2753.933551] Key type ._llcrypt registered [ 2753.934884] Key type .llcrypt registered [ 2753.971640] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 2754.066438] Key type .llcrypt unregistered [ 2754.067487] Key type ._llcrypt unregistered [ 2760.383666] Key type ._llcrypt registered [ 2760.384895] Key type .llcrypt registered [ 2760.419431] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 2760.711333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2760.734000] alg: No test for adler32 (adler32-zlib) [ 2761.575710] Lustre: Lustre: Build Version: 2.17.54_105_g95cff68 [ 2761.653813] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 2763.231082] Key type lgssc registered [ 2763.526759] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2765.930142] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2765.937071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2781.343429] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2786.463335] LustreError: 185429:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.157@tcp: failed processing log, type 4: rc = -110 [ 2813.087202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2814.964830] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2817.318534] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2819.508332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2819.683501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2819.708822] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2821.018269] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2823.801078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2823.937158] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2825.313205] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2827.672408] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2829.484714] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2830.432345] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 2838.497897] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2838.502410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2838.504085] Lustre: Skipped 1 previous similar message [ 2841.296341] Lustre: server umount lustre-OST0000 complete [ 2843.616942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2843.617870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2843.620820] Lustre: Skipped 2 previous similar messages [ 2843.624248] Lustre: Skipped 1 previous similar message [ 2848.734301] Lustre: server umount lustre-MDT0000 complete [ 2848.736136] LustreError: 186468:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2848.742823] LustreError: 186468:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2849.988351] LustreError: 187190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783519652 with bad export cookie 12581979794202012411 [ 2849.991064] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2849.994351] LustreError: 187190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2850.116555] Lustre: server umount lustre-MDT0001 complete [ 2853.793839] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 2854.781312] Key type lgssc unregistered [ 2854.916393] LNet: 189342:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2854.920190] LNetError: 189342:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2854.931509] LNet: Removed LNI 192.168.203.157@tcp [ 2855.252221] Key type .llcrypt unregistered [ 2855.253362] Key type ._llcrypt unregistered [ 2864.230318] Key type ._llcrypt registered [ 2864.231944] Key type .llcrypt registered [ 2864.280654] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 2864.716739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2864.767764] alg: No test for adler32 (adler32-zlib) [ 2865.633731] Lustre: Lustre: Build Version: 2.17.54_105_g95cff68 [ 2865.734983] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 2867.327151] Key type lgssc registered [ 2867.740195] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2870.461923] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2870.468295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2871.619791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2872.987438] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2875.765432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2875.895684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2877.241476] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2879.561302] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2881.308650] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2883.456448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2883.570714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2885.426963] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2887.782102] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2896.869779] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 2899.348343] Lustre: Failing over lustre-MDT0000 [ 2899.433756] Lustre: server umount lustre-MDT0000 complete [ 2901.471795] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2901.474403] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2901.478936] LustreError: 190729:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2901.486930] LustreError: 190729:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2901.984974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2901.986060] LustreError: 190725:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2901.988794] Lustre: Skipped 1 previous similar message [ 2903.284117] LustreError: 190726:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2903.289346] LustreError: 190726:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2907.104161] LustreError: 190748:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2907.108821] LustreError: 190748:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2912.224124] LustreError: 190730:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2912.228812] LustreError: 190730:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2912.317040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2912.355688] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2912.452081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2912.475038] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2913.518130] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2913.942912] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2915.630168] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 2917.862968] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2917.865276] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2917.882238] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 2920.203615] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 2926.560131] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2926.562832] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2926.570570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2928.097125] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2928.098100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2928.102873] Lustre: Skipped 1 previous similar message [ 2932.561541] Lustre: server umount lustre-OST0000 complete [ 2938.336871] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2938.336879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2938.337494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2938.340449] Lustre: Skipped 2 previous similar messages [ 2938.348562] Lustre: Skipped 1 previous similar message [ 2940.059268] Lustre: server umount lustre-MDT0000 complete [ 2941.338470] LustreError: 190710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783519743 with bad export cookie 11291840927190664555 [ 2941.342367] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2941.342964] LustreError: 190710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2941.453024] Lustre: server umount lustre-MDT0001 complete [ 2945.144316] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 2946.180739] Key type lgssc unregistered [ 2946.308361] LNet: 195626:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2946.310899] LNetError: 195626:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2946.318311] LNet: Removed LNI 192.168.203.157@tcp [ 2946.621511] Key type .llcrypt unregistered [ 2946.622508] Key type ._llcrypt unregistered [ 2955.810854] Key type ._llcrypt registered [ 2955.811884] Key type .llcrypt registered [ 2955.850170] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 2956.135783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2956.142543] alg: No test for adler32 (adler32-zlib) [ 2957.005734] Lustre: Lustre: Build Version: 2.17.54_105_g95cff68 [ 2957.101894] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 2958.687158] Key type lgssc registered [ 2959.055547] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2961.641616] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2961.648556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2962.826148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2964.262206] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2967.032925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2967.143821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2968.437677] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2970.705281] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2972.441553] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2974.665876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2974.786902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2976.660407] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2979.049076] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2989.158774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 2990.467884] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 2997.029315] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 3008.614452] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3018.208387] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3018.213390] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3018.219337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3020.258824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3020.259074] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3020.259301] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3022.875932] Lustre: server umount lustre-OST0000 complete [ 3025.376671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3025.377547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3025.380171] Lustre: Skipped 1 previous similar message [ 3025.384222] Lustre: Skipped 2 previous similar messages [ 3030.228711] Lustre: server umount lustre-MDT0000 complete [ 3030.496041] LustreError: 197013:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3030.502516] LustreError: 197013:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3031.473201] LustreError: 196995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783519834 with bad export cookie 2693469651022376457 [ 3031.476102] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3031.479733] LustreError: 196995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3031.603402] Lustre: server umount lustre-MDT0001 complete [ 3035.269179] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 3036.299501] Key type lgssc unregistered [ 3036.424319] LNet: 200881:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3036.427147] LNetError: 200881:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3036.436301] LNet: Removed LNI 192.168.203.157@tcp [ 3036.751641] Key type .llcrypt unregistered [ 3036.752989] Key type ._llcrypt unregistered [ 3043.779511] Key type ._llcrypt registered [ 3043.780851] Key type .llcrypt registered [ 3043.815410] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 3044.097114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3044.127069] alg: No test for adler32 (adler32-zlib) [ 3044.970571] Lustre: Lustre: Build Version: 2.17.54_105_g95cff68 [ 3045.048763] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 3046.623134] Key type lgssc registered [ 3046.917155] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3049.283364] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3049.289548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3050.431181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3051.686860] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3054.396158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3054.509927] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3055.718738] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3058.124705] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3060.029450] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3062.383845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3062.510736] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3064.395045] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3066.766224] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3075.811753] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 3078.796368] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 3087.654862] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3096.827791] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 3105.631060] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 3116.000374] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3116.004060] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3116.009936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3117.025042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3117.026019] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3117.030777] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3122.145084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3122.147875] Lustre: Skipped 2 previous similar messages [ 3127.265368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3127.267202] Lustre: Skipped 1 previous similar message [ 3130.335115] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3130.392232] Lustre: server umount lustre-OST0000 complete [ 3132.385156] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3132.390196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3132.392331] Lustre: Skipped 1 previous similar message [ 3137.817819] Lustre: server umount lustre-MDT0000 complete [ 3139.014173] LustreError: 205316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783519941 with bad export cookie 17493327814685962693 [ 3139.016654] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3139.018119] LustreError: 205316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3139.144971] Lustre: server umount lustre-MDT0001 complete [ 3142.730511] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 3143.706529] Key type lgssc unregistered [ 3143.827330] LNet: 207789:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3143.829480] LNetError: 207789:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3143.838286] LNet: Removed LNI 192.168.203.157@tcp [ 3144.137303] Key type .llcrypt unregistered [ 3144.138431] Key type ._llcrypt unregistered [ 3152.750378] Key type ._llcrypt registered [ 3152.751597] Key type .llcrypt registered [ 3152.789184] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 3153.259820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3153.273273] alg: No test for adler32 (adler32-zlib) [ 3154.120126] Lustre: Lustre: Build Version: 2.17.54_105_g95cff68 [ 3154.201460] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 3155.775090] Key type lgssc registered [ 3156.078846] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3158.376526] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3158.382898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3159.512831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3160.697354] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3163.231067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3163.359334] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3164.573645] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3166.679300] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3168.369275] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3170.697649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3170.832706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3172.547514] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3174.752407] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3185.190531] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 3190.244597] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3190.245235] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3190.249046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3190.253894] Lustre: Skipped 1 previous similar message [ 3194.773839] Lustre: server umount lustre-OST0000 complete [ 3199.455919] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3199.458970] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3199.462310] Lustre: Skipped 1 previous similar message [ 3199.464065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3200.480171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3200.480886] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3202.195615] Lustre: server umount lustre-MDT0000 complete [ 3203.418278] LustreError: 209158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783520006 with bad export cookie 17561187039454144305 [ 3203.422058] LustreError: 209158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3203.423620] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3203.530964] Lustre: server umount lustre-MDT0001 complete [ 3207.113648] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 3208.183723] Key type lgssc unregistered [ 3208.308387] LNet: 212880:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3208.311550] LNetError: 212880:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3208.322329] LNet: Removed LNI 192.168.203.157@tcp [ 3208.633436] Key type .llcrypt unregistered [ 3208.634600] Key type ._llcrypt unregistered [ 3215.995000] Key type ._llcrypt registered [ 3215.996105] Key type .llcrypt registered [ 3216.038748] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 3216.461415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3216.484545] alg: No test for adler32 (adler32-zlib) [ 3217.325847] Lustre: Lustre: Build Version: 2.17.54_105_g95cff68 [ 3217.400959] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 3218.975158] Key type lgssc registered [ 3219.303969] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3221.696211] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3221.701065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3222.826391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3224.160962] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3227.010863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3227.146687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3228.928721] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3232.982840] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3238.497595] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3247.521983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3247.625947] LustreError: 215943:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3247.646775] LustreError: 215943:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3248.009986] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3253.172715] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 3256.197274] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3265.435615] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3278.818726] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3278.832043] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3278.850295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3278.859862] Lustre: Skipped 1 previous similar message [ 3278.886569] Lustre: Skipped 1 previous similar message [ 3283.936123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3283.941931] Lustre: Skipped 1 previous similar message [ 3284.973963] Lustre: server umount lustre-OST0000 complete [ 3288.544474] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3288.549962] LustreError: Skipped 1 previous similar message [ 3288.555128] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3288.568464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3294.188417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3294.197505] Lustre: Skipped 3 previous similar messages [ 3294.524708] Lustre: server umount lustre-MDT0000 complete [ 3296.573091] LustreError: 214104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783520099 with bad export cookie 10835189882442685315 [ 3296.578016] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3296.580477] LustreError: 214104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3296.752039] Lustre: server umount lustre-MDT0001 complete [ 3301.553572] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 3302.802252] Key type lgssc unregistered [ 3302.936609] LNet: 217802:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3302.939710] LNetError: 217802:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3302.949468] LNet: Removed LNI 192.168.203.157@tcp [ 3303.332682] Key type .llcrypt unregistered [ 3303.334054] Key type ._llcrypt unregistered [ 3312.550766] Key type ._llcrypt registered [ 3312.552072] Key type .llcrypt registered [ 3312.606207] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 3313.114108] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3313.123088] alg: No test for adler32 (adler32-zlib) [ 3314.030663] Lustre: Lustre: Build Version: 2.17.54_105_g95cff68 [ 3314.147796] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 3315.767359] Key type lgssc registered [ 3316.274634] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3320.075936] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3320.088669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3321.296769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3323.131489] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3327.319435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3327.534775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3329.494959] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3333.255730] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3336.023248] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3339.820653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3340.041470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3343.287922] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3346.219975] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 3347.418630] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3353.381128] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3353.427195] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3353.546328] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3356.464748] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3359.928496] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3360.627669] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3371.529912] Lustre: Permanently deactivating lustre-OST0001 [ 3371.538390] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3380.253852] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3382.914830] Lustre: Permanently reactivating lustre-OST0001 [ 3382.924627] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 3382.930924] Lustre: Skipped 2 previous similar messages [ 3387.420906] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3387.432320] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3387.439961] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3387.452418] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3387.489743] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3391.459727] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3391.469648] Lustre: Skipped 1 previous similar message [ 3391.473470] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3391.478597] Lustre: Skipped 1 previous similar message [ 3396.558122] Lustre: server umount lustre-OST0001 complete [ 3396.582911] LustreError: 220863:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3396.594394] LustreError: 220863:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3398.627092] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3398.630443] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3398.634761] Lustre: Skipped 1 previous similar message [ 3398.638487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3401.697107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3401.698049] LustreError: 220864:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3401.700871] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3401.704577] Lustre: Skipped 1 previous similar message [ 3401.730307] LustreError: 220864:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3404.758433] Lustre: server umount lustre-OST0000 complete [ 3409.375983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3409.383723] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3409.394055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3413.193363] Lustre: server umount lustre-MDT0000 complete [ 3415.404502] LustreError: 221882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783520218 with bad export cookie 13683412335937839383 [ 3415.412386] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3415.416107] LustreError: 221882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3415.649517] Lustre: server umount lustre-MDT0001 complete [ 3421.064743] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 3422.379723] Key type lgssc unregistered [ 3422.527459] LNet: 224085:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3422.531558] LNetError: 224085:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3422.544738] LNet: Removed LNI 192.168.203.157@tcp [ 3422.938074] Key type .llcrypt unregistered [ 3422.939277] Key type ._llcrypt unregistered [ 3442.239189] Key type ._llcrypt registered [ 3442.242990] Key type .llcrypt registered [ 3442.306708] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 3442.742455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3442.791051] alg: No test for adler32 (adler32-zlib) [ 3443.688208] Lustre: Lustre: Build Version: 2.17.54_105_g95cff68 [ 3443.809662] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 3445.431155] Key type lgssc registered [ 3446.067573] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3450.336747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3450.347125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3451.465918] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3451.477740] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3451.557713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3453.501657] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3457.487707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3457.542933] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3457.558365] Lustre: 226266:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3457.700361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3459.505469] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3463.020953] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3465.975830] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3469.384594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3469.494133] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3469.609996] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3472.795658] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3476.871743] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3478.577945] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 3483.986041] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 3487.635593] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 3496.564463] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 3502.424461] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3512.367279] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3535.893835] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 3535.898524] Lustre: Skipped 2 previous similar messages [ 3570.630885] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3570.635376] Lustre: Skipped 3 previous similar messages [ 3640.216401] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 3640.219922] Lustre: Skipped 9 previous similar messages [ 3645.926595] Lustre: server umount lustre-OST0000 complete [ 3652.575865] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3652.579086] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3652.581745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3652.588425] Lustre: Skipped 1 previous similar message [ 3657.698529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3657.702278] Lustre: Skipped 2 previous similar messages [ 3657.746022] Lustre: server umount lustre-MDT0000 complete [ 3659.688649] LustreError: 228892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783520462 with bad export cookie 18013480399567571020 [ 3659.705817] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3670.075205] Lustre: server umount lustre-MDT0001 complete [ 3675.686231] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 3677.471811] Key type lgssc unregistered [ 3677.641726] LNet: 231051:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3677.650350] LNetError: 231051:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3677.660751] LNet: Removed LNI 192.168.203.157@tcp [ 3678.105511] Key type .llcrypt unregistered [ 3678.107461] Key type ._llcrypt unregistered [ 3689.174460] Key type ._llcrypt registered [ 3689.175867] Key type .llcrypt registered [ 3689.218375] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 3689.576324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3689.625138] alg: No test for adler32 (adler32-zlib) [ 3690.506529] Lustre: Lustre: Build Version: 2.17.54_105_g95cff68 [ 3690.612612] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 3692.207693] Key type lgssc registered [ 3692.681553] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3696.247980] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3696.261879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3697.450715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3699.206819] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3703.429825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3703.637805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3705.663821] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3709.168388] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3711.615585] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3715.009232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3715.218697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3717.947982] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3721.697510] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3722.411336] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 3743.200324] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3743.203397] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3743.208363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3744.227700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3744.231135] Lustre: Skipped 2 previous similar messages [ 3745.583120] Lustre: server umount lustre-OST0000 complete [ 3749.344714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3749.347182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3749.352974] Lustre: Skipped 2 previous similar messages [ 3753.383400] Lustre: server umount lustre-MDT0000 complete [ 3754.970149] LustreError: 232423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783520557 with bad export cookie 10912328544671064592 [ 3754.971273] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3754.981607] LustreError: 232423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3755.129666] Lustre: server umount lustre-MDT0001 complete [ 3759.670533] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 3760.975692] Key type lgssc unregistered [ 3761.151619] LNet: 236137:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3761.157870] LNetError: 236137:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3761.171654] LNet: Removed LNI 192.168.203.157@tcp [ 3761.589551] Key type .llcrypt unregistered [ 3761.591366] Key type ._llcrypt unregistered [ 3771.657867] Key type ._llcrypt registered [ 3771.659255] Key type .llcrypt registered [ 3771.703767] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 3772.145189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3772.182868] alg: No test for adler32 (adler32-zlib) [ 3773.068072] Lustre: Lustre: Build Version: 2.17.54_105_g95cff68 [ 3773.173368] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 3774.767385] Key type lgssc registered [ 3775.237248] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4041.323551] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 4042.444236] Key type lgssc unregistered [ 4042.590676] LNet: 238137:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4042.593381] LNetError: 238137:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4042.605543] LNet: Removed LNI 192.168.203.157@tcp [ 4042.949775] Key type .llcrypt unregistered [ 4042.951509] Key type ._llcrypt unregistered [ 4052.359043] Key type ._llcrypt registered [ 4052.360608] Key type .llcrypt registered [ 4052.404671] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [ 4059.732273] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 4060.052110] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4060.083968] alg: No test for adler32 (adler32-zlib) [ 4060.972161] Lustre: Lustre: Build Version: 2.17.54_105_g95cff68 [ 4061.078338] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 4062.679157] Key type lgssc registered [ 4063.151924] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4075.170485] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 3953 sec ========== 10:27:57 (1783520877) [ 4075.891489] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 10:27:57 (1783520877) === [ 4077.283887] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 10:27:59 (1783520879) === [ 4092.768272] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 4093.970854] Key type lgssc unregistered [ 4094.113316] LNet: 243343:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4094.116311] LNetError: 243343:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4094.126271] LNet: Removed LNI 192.168.203.157@tcp [ 4094.464471] Key type .llcrypt unregistered [ 4094.466046] Key type ._llcrypt unregistered